Theories of rhetoric and architecture suggest that buildings designed to be high ranking according to the Western architectural decorum have more impact on the minds of their beholders than low-ranking buildings. Here, we used event-related potentials in a visual object categorization task to probe this assumption and to examine whether the hippocampus contributes to the processing of architectural ranking. We found that early negative potentials between 200 and 400 ms differentiated between high- and low-ranking buildings in healthy subjects and patients with temporal lobe epilepsy with and without hippocampal sclerosis. By contrast, late positive potentials between 400 and 600 ms were higher in amplitude to high-ranking buildings only in healthy subjects and TLE patients without but not in TLE patients with hippocampal sclerosis. These findings suggest that the differentiation between high- and low-ranking buildings entails both early visual object selection and late post-model selection processes and that the hippocampus proper contributes critically to this second stage of visual object categorization.