Rare pathogenic variants in known breast cancer susceptibility genes and known common susceptibility variants do not fully explain the familial aggregation of breast cancer. To investigate plausible genetic models for the residual familial aggregation, we studied 17,425 families ascertained through population-based probands, 86% of whom were screened for pathogenic variants in BRCA1, BRCA2, PALB2, CHEK2, ATM and TP53 using gene-panel sequencing. We conducted complex segregation analyses and fitted genetic models in which breast cancer incidence depended on the effects of pathogenic variants in known susceptibility genes and other unidentified major genes, and a normally distributed polygenic component. The proportion of familial variance explained by BRCA1, BRCA2, PALB2, CHEK2, ATM and TP53 was 46% at age 20-29 years and decreased steadily with age thereafter. After allowing for these genes, the best fitting model for the residual familial variance included a recessively inherited risk component with a combined genotype frequency of 1.7% (95% CI: 0.3-5.4%) and a penetrance to age 80 years of 69% (95% CI: 38-95%) for homozygotes, and a polygenic variance of 1.27 (95% CI: 0.94-1.65) which did not vary with age. The proportion of the residual familial variance explained by the recessive risk component was 40% at age 20-29 years and decreased with age thereafter. The model predicted age-specific familial relative risks consistent with those observed by large epidemiological studies. The findings have implications for strategies to identify new breast cancer susceptibility genes and improve breast cancer risk prediction, especially at a young age.