"The dress" has provoked intensive commentary among psychophysicists, especially in relation to color vision. Researchers have shown that manipulating illuminance cues can influence the perceived colors of the dress. Here we investigate whether illusory shifts in brightness can shift color perception of the dress. Drifting achromatic gratings with fast off and fast on shading profiles are known to give an illusion of brightening or darkening, respectively. We superimposed rotating sawtooth gratings on a series of dress images that morphed from extreme white/gold through to blue/black. In a sample of 18 adults (11 with white/gold dress percept and seven with blue/black percept), a two-alternative, forced-choice constant stimulus task measured the morphed image point at which each observer was equally likely to categorize the dress as white/gold or blue/black (the point of subjective equality or PSE). Despite manifest individual differences in the PSE, the two sawtooth temporal profiles consistently changed the perceived colors of the dress. Perceptual dimming shifted color categorization toward blue/black whereas perceptual brightening shifted color categorization toward white/gold. We conclude that color categorization is influenced substantially by illusory shifts in brightness.