Risk adjustment is used widely in payment systems and performance assessments, but the extent to which it distinguishes plan or provider effects from confounding due to patient differences is typically unknown.To assess the degree to which risk-adjusted measures of health plan performance adequately adjust for the variation across plans that arises because of differences in patient characteristics (residual confounding).Comparison between plan performance estimates based on enrollees who made plan choices (observational population) and estimates based on enrollees assigned to plans (randomized population).Natural experiment in which more than two thirds of a state's Medicaid population in 1 region was randomly assigned to 1 of 5 plans.137 933 enrollees in 2013 to 2014, of whom 31.1% selected a plan and 68.9% were randomly assigned to 1 of the same 5 plans.Annual total spending (that is, payments to providers), primary care use, dental care use, and avoidable emergency department visits, all scored as plan-specific deviations from the "average" plan performance within each population.Enrollee characteristics were appreciably imbalanced across plans in the observational population, as expected, but were not in the randomized population. Annual total spending varied across plans more in the observational population (SD, $147 per enrollee) than in the randomized population (SD, $70 per enrollee) after accounting for baseline differences in the observational and randomized populations and for differences across plans. On average, a plan's spending score (its deviation from the "average" performance) in the observational population differed from its score in the randomized population by $67 per enrollee in absolute value (95% CI, $38 to $123), or 4.2% of mean spending per enrollee (Potential heterogeneity in plan effects between the 2 populations.Residual confounding in risk-adjusted performance assessments can be substantial and should caution policymakers against assuming that risk adjustment isolates real differences in plan performance.Arnold Ventures.