Natural illumination, realistic specular reflections, and complex cast shadows are key features in determining the appearance of objects. However, they have largely been ignored in computer vision algorithms. Indeed, most algorithms assume Lambertian objects lit by a single point source, without any explicit notion of visibility. Our goal is to derive the theoretical foundations and mathematical equations and computational techniques to allow much more robust computer vision algorithms that can work in outdoor lighting conditions, with difficult specular objects, taking shadows and interreflections into account.
Primary Participants 
Publications 
The theoretical results for the Lambertian case , and especially the 9 parameter model have been widely used to handle complex illumination in many areas of vision, including lightinginsensitive recognition, structure from motion, and photometric stereo. Recently, we have linked the convolution results with principal component analysis, deriving more compact subspaces and explaining some classic previous empirical results. Our results enable complex illumination on Lambertian matte surfaces to be handled with essentially the same complexity as point light sources. In more recent work, we have also developed a compact model of specularities, combining that with the Lambertian results to use specularities for the first time as a positive source of information in lightinginsensitive recognition. We have recently also looked at cast shadows, especially in natural 3D textures like moss, sponge or gravel, showing that in many canonical cases, a convolution result can be derived and used to explain some observations regarding lighting variability. A good overview of spherical harmonic lighting and the theoretical basis behind our methods is given in a recent book chapter. We have also derived a new class of frequency domain invariants or spherical harmonic identities . They can be used for direct transferbased relighting, without formal inverse rendering, and to check the consistency of images to detect tampering or splicing. Most recently, we have developed a new characterization of inverse light transport, a new differential theory of surface reconstruction for general reflectance, and formal conditions for reflectance and light transport estimation.

Analytic
PCA Construction for Theoretical Analysis of Lighting Variability,
Including Attached Shadows, in a Single Image of a Convex Lambertian
Object PAMI Oct 2002,
pp 13221333.
We explain for the first time some classic empirical results on lighting variability, and take a first step toward analyzing many classic vision problems under complex lighting. Full Paper: PDF (.8M) 

Using Specularities for Recognition ICCV 03, pages 15121519 We present the first method for using specularities as a positive feature for lightinginsensitive recognition. The method is applied to very difficult objects like shiny crockery and wine glasses. Paper: PDF 

A Fourier Theory for Cast Shadows ECCV 04, pages I 146162 ; PAMI Feb 05, pages 288295 We show that cast shadows can be mathematically analyzed for many simple configurations, resulting in a standard convolution formula that can be derived analytically in 2D and analyzed numerically in 3D. The results help explain many effects of lighting variability in 3D textures and suggest new bases for that purpose. Paper: ECCV 04 , PAMI 05 

Modeling Illumination Variation with Spherical Harmonics
Book chapter in Face Processing: Advanced Modeling Methods (pages 385424, 2005) The appearance of objects including human faces can vary dramatically with the lighting. We present results that use spherical harmonic illumination basis functions to understand this variation for face modeling and recognition, as well as a number of other applications in graphics and vision. Paper: PDF 

A Theory of Frequency Domain Invariants: Spherical Harmonic Identities for BRDF/Lighting
Transfer and Image Consistency
ECCV 06, vol IV, pp 4155, PAMI 30(2), pages 197213, Feb 2008. We develop new mathematical results based on the spherical harmonic convolution framework for reflection, deriving novel identities, which are the angular frequency domain analogs to common spatial domain invariants such as reflectance ratios. Paper: PDF (PAMI 08) PDF (ECCV 06) 
On the Duality of Forward and Inverse Light Transport
PAMI 2011, ECCV 2010 Inverse light transport seeks to undo global illumination effects, such as interreections, that pervade images of most scenes. This paper presents the theoretical and computational foundations for inverse light transport as a dual of forward rendering. We demonstrate two practical applications, namely, separation of individual bounces of the light transport and fast projector radiometric compensation to display images free of global illumination artifacts in realworld environments. Paper: PAMI ECCV Tech Report Video 

What an Image Reveals About Material Reflectance
ICCV 2011 We derive precise conditions under which material reflectance properties may be estimated from a single image of a homogeneous curved surface (canonically a sphere), lit by a directional source. Based on the observation that light is reflected along certain (a priori unknown) preferred directions such as the halfangle, we propose a semiparametric BRDF abstraction that lies between purely parametric and purely datadriven models. While it is wellknown that fitting multilobe BRDFs may be illposed under certain conditions, prior to this work, precise results for the wellposedness of BRDF estimation had remained elusive. Paper: PDF 

From the Rendering Equation to Stratified Light Transport Inversion International Journal of Computer Vision, 2012 In this work, we explore a theoretical analysis of inverse light transport, relating it to its forward counterpart, expressed in the form of the rendering equation. We show the existence of an inverse Neumann series, that zeroes out the corresponding physical bounces of light, which we refer to as stratified light transport inversion. Our practical application is to radiometric compensation, where we seek to project patterns onto realworld surfaces, undoing the effects of global illumination. Paper: PDF 

FrequencySpace Decomposition and Acquisition of Light Transport under
Spatially Varying Illumination
ECCV 2012. We show that, under spatially varying illumination, the light transport of diffuse scenes can be decomposed into direct, nearrange (subsurface scattering and local interreflections) and far range transports (diffuse interreflections). We show that these three component transports are redundant either in the spatial or the frequency domain and can be separated using appropriate illumination patterns, achieving a theoretical lower bound. Paper: PDF Talk: PPT 

Sharpening Out of Focus Images using HighFrequency Transfer
EuroGraphics 2013. We propose a new method to sharpen outoffocus images, that uses a similar but different assisting sharp image provided by the user (such as multiple images of the same subject in different positions captured using a burst of photographs). We demonstrate sharpened results on outoffocus images in macro, sports, portrait and wildlife photography. Paper: PDF Video (MPEG) Supplementary HTML 