ObjectivesThe use of external control arms to study treatment effects is growing in interest among drug sponsors and regulators. However, experience with performing these kinds of studies for complex, immune-mediated diseases is limited. We sought to analyze a retrospective cohort of Crohn’s patients to predict the outcome of a prospective cohort.MethodsWe queried electronic health records databases and screened records at the University of California, San Francisco to identify patients meeting the eligibility criteria of TRIDENT, a concurrent trial involving ustekinumab as a reference arm. Timepoints were defined to balance the tradeoff between missing disease activity and bias. We compared two imputation models by their impacts on cohort membership and outcomes. We compared the results of ascertaining disease activity using structured data algorithms against manual review. We used these data to estimate ustekinumab’s real-world effectiveness.ResultsScreening identified 183 patients. 30% of the cohort had missing baseline data. Two imputation models were tested and had similar effects on cohort definition and outcomes. Algorithms for ascertaining non-symptom-based elements of disease activity were similar in accuracy to manual review. The final cohort consisted of 56 patients. 34% of the cohort was in steroid-free clinical remission by week 24.ConclusionsWe predict that a third of the ustekinumab-treated cohort in TRIDENT will be in steroid-free remission by week 24. However, our prediction is limited by substantial missing data. Efforts to improve real-world data capture and align trial design with clinical practice may enable more robust future studies and improve trial efficiency.STUDY HIGHLIGHTSWHAT IS KNOWNExternal control arm studies are receiving growing interest from drug sponsors and regulators as a potential source of real-world evidenceHowever, the feasibility and robustness of this approach is currently limitedUstekinumab is an FDA-approved treatment of moderately to severely active Crohn’s diseaseWHAT IS NEW HEREWe derived a retrospective cohort of patients designed to resemble the participants of TRIDENT, a concurrent phase 2b trial using ustekinumab as a reference armWe predict that about 34% of the ustekinumab-assigned participants in TRIDENT will be in steroid-free clinical remission by week 24.EHR structured data algorithms may be an accurate and less laborious alternative to manual abstraction of non-symptom-based components of the Crohn’s Disease Activity IndexReal-world practice differs from controlled studies in important ways, including the treatment goals and the timing of encountersThese differences can pose challenges to the feasibility of external control arm studies, and must be addressed to enable this novel study designThe methods, data, and code used in this pilot study are shared here for reproducibility and enhancement by others