The problem of cross-domain face recognition seeks to identify facial images obtained from different domains, and it is gaining popularity due to its numerous applications in law enforcement identification and camera surveillance Existing algorithms typically fail to fully exploit semantic information for identifying crossdomain faces, which could be a strong clue for recognition. In this paper, we present an efficient algorithm for cross-domain face recognition that makes use of semantic information in conjunction with deep convolutional neural networks (CNN). We start with a soft face parsing algorithm that measures the boundaries of facial components as probabilistic values. For cross domain face recognition, we propose a hierarchical soft semantic representation framework. CNN-derived deep features are computed and combined. Which could fully exploit the same semantic clue across cross-domain faces. We present extensive experiments to show that the proposed soft semantic representation algorithm outperforms state-of-the-art methods.