Network embedding, as an effective method of learning the low-dimensional representations of nodes, has been widely applied to various complex network analysis tasks, such as node classification, community detection, link prediction and evolution analysis. The existing embedding methods usually focus on the local structure of the network by capturing community structure, first-order or second-order proximity, etc. Some methods have been proposed to model the high-order proximity of networks to capture more effective information. However, they are incapable of preserving the similarity among nodes that are not very close to each other in network but have similar structures. For instance, the nodes with similar local topology structure should be similar in embedding space even if they are not in the same community. Herein, we regard these structure characteristics as the high-order features, which reveals that the structure similarity between nodes is spatially unrelated. In light of above the limitations of existing methods, we construct the high-order feature matrix for mutually reinforcing the embedding which preserves the local structure. To integrate these features effectively, we propose LHO-NMF, which fuses the high-order features into non-negative matrix factorization framework while capturing the local structure. The proposed LHO-NMF could effectively learn the node representations via preserving the local structure and high-order feature information. In specific, the high-order features are learned based on random walk algorithm. The experimental results show that the proposed LHO-NMF method is very effective and outperforms other state-of-the-art methods among multiple downstream tasks.