Objective. Neuroimaging uncovers important information about disease in the brain. Yet in Alzheimer’s disease (AD), there remains a clear clinical need for reliable tools to extract diagnoses from neuroimages. Significant work has been done to develop deep learning (DL) networks using neuroimaging for AD diagnosis. However, no particular model has emerged as optimal. Due to a lack of direct comparisons and evaluations on independent data, there is no consensus on which modality is best for diagnostic models or whether longitudinal information enhances performance. The purpose of this work was (1) to develop a generalizable DL model to distinguish neuroimaging scans of AD patients from controls and (2) to evaluate the influence of imaging modality and longitudinal data on performance. Approach. We trained a 2-class convolutional neural network (CNN) with and without a cascaded recurrent neural network (RNN). We used datasets of 772 (N AD = 364, N control = 408) 3D 18F-FDG PET scans and 780 (N AD = 280, N control = 500) T1-weighted volumetric-3D MR images (containing 131 and 144 patients with multiple timepoints) from the Alzheimer’s Disease Neuroimaging Initiative, plus an independent set of 104 (N AD = 63, N NC = 41) 18F-FDG PET scans (one per patient) for validation. Main Results. ROC analysis showed that PET-trained models outperformed MRI-trained, achieving maximum AUC with the CNN + RNN model of 0.93 ± 0.08, with accuracy 82.5 ± 8.9%. Adding longitudinal information offered significant improvement to performance on 18F-FDG PET, but not on T1-MRI. CNN model validation with an independent 18F-FDG PET dataset achieved AUC of 0.99. Layer-wise relevance propagation heatmaps added CNN interpretability. Significance. The development of a high-performing tool for AD diagnosis, with the direct evaluation of key influences, reveals the advantage of using 18F-FDG PET and longitudinal data over MRI and single timepoint analysis. This has significant implications for the potential of neuroimaging for future research on AD diagnosis and clinical management of suspected AD patients.