Land application of animal manure has been encouraged widely in China. This presents a risk of heavy metals (HMs) accumulation in the soil due to their high contents in the feeds and additives. A 4-year field-scale study was conducted to monitor and estimate HMs accumulation in the soil with land application of swine manure. The results show a clear tendency for As, Hg, Cr, Cu, Zn and Mn to increase gradually with the application duration, yielding an average annual increase of 0.57, 0.011, 6.20, 5.64, 22.58, and 23.45 mg kg−1, respectively, at the annual application rate of about 250 t ha−1 of swine manure. The estimation from the mass balance modeling indicates the environmental risk of Cd, Cu and Zn will exceed the threshold levels for agricultural soils in China in the next 10–50 years. Determination of a suitable application rate of animal manure would be the first consideration for mitigating the environmental risk of HMs currently.