Estimating orientation field for latent fingerprints plays a crucial role in latent fingerprints recognition systems. Due to poor quality and small area of latent fingerprints, however, the performance of the state-of-the-art algorithms is still far from satisfactory. Considering the intrinsic characteristics of fingerprints that the distribution of orientation field varies with the fingerprint patterns, we propose an orientation field estimation algorithm for latent fingerprints based on residual learning using prior knowledge of fingerprint patterns. Specifically, statistical distribution models of orientation field, for different fingerprint patterns, are calculated based on a large database consisting of 14,000 fingerprints with good quality using clustering method. The residual orientation fields and reliability scores, indicating the consistency with different statistical orientation models, are estimated using a deep network, named RefNet. Then the final orientation field is obtained by fusing the estimations according to their corresponding reliability scores. Experimental results on the widely used latent database NIST SD27 demonstrate that the proposed algorithm provides higher orientation field estimation accuracy compared with the state-of-the-art methods, and by enhancing latent fingerprints using estimated orientation field, the identification performance is further improved.