We present a method that extends techniques in [Yu and Malik 1998] and [Debevec 1998] to estimate the surface colors of a complex scene with diffuse surfaces lit by natural outdoor illumination. Given a model of the scene’s geometry, a set of photographs of the scene taken under natural illumination, and corresponding measurements of the illumination, we can calculate the spatially-varying diffuse surface reflectance. The process employs a simple iterative inverse global illumination technique to compute the surface colors for the scene which, when rendered under the recorded illumination, best reproduce the appearance in the photographs. The results can then be used to render the scene under novel illumination.