Single-Image Reflectance Estimation for Relighting by Iterative Soft GroupingYuanzhen Li1,2Stephen Lin1Sing Bing Kang3Hanqing Lu2Heung-Yeung Shum11Microsoft Research, Asia2Chinese Academy of Sciences3Microsoft ResearchAbstractReflectance values for image-based relighting are oftenestimated from grouped pixels with similar reflectance, butsuch groupings are difficult to compute with certainty for sparse image data. To address this problem, we propose an iterative method that aggregates BRDF data in a single image with known geometry and lighting by soft grouping, where pixels contribute to one anothers estimate accordingto their degree of reflectance similarity. Estimation of spec-ular reflectance is further improved by albedo-independent soft grouping of pixels based on shape continuity. With re-covered reflectances, we demonstrate realistic relighting for synthetic and real scenes, including surfaces with spatially-varying reflectance.1. IntroductionPhotographs have commonly been used in computer graphics for realistic rendering of scenes under various il- lumination conditions. Previous approaches to relighting scenes include capturing a set of images under densely sam- pled lighting directions 2, or estimating parameters of re-flectance models, which can be used for rendering with ar- bitrary lighting 6910. To avoid the need for large sets of images and multi- ple illumination conditions, some previous methods processbidirectional reflectance distribution function (BRDF) dataaggregated from predefined groupings of pixels with similarreflectance 1181.While these methods assume a prior reflectance group-ing, formation of these groups for reflectance estimationis a difficult task, especially when local reflectance vari- ations often exist.To address this problem, Lensch etal. 4 present a method for clustering reflectances by it-erative splitting and refitting of reflectance models to auser-specified number of materials. Spatially-varying re-flectances are preserved by expressing the BRDF of each pixel in terms of basis BRDFs of its cluster. Nishino et al.7 assemble reflectance data from point correspondencesamong multiple views with fixed lighting to estimate re-flectance parameters as well as the illumination environ- ment. In our work, the goal is also to aggregate BRDF infor-mation for reflectance estimation, but with substantially re- duced data. For broader applicability, our approach takes as input only a single image, and with such limited data, we cannot perform clustering as in 4 where 20-25 images arecaptured. In fact, a single image is insufficient for estimat-ing the reflectance of independent pixels, so some form of data aggregation becomes necessary. Determination of pixel groupings is, however, a chal- lenging problem. With just a single BRDF sample for each pixel, it cannot be known with full certainty whether twopixels share the same reflectance. Furthermore, the pres- ence of spatially-varying BRDFs will complicate the par- titioning of pixels. To deal with this uncertainty, we pro-pose a soft reflectance grouping where pixels contribute invarying degrees to one anothers reflectance estimate. Incomputing the reflectance of a given pixel, the BRDF data of other neighboring pixels are each weighted by their re-flectance similarity to the examined pixel, so that pixelswhich are more likely of the same reflectance are more strongly grouped while less similar pixels have relatively little impact on the estimation. By computing these soft groupings separately for each pixel, spatially-varying re-flectance is also modelled in this framework. Because of initially imprecise soft groupings, we pro-gressively refine reflectance estimates by iterating the pro- cess. In successive iterations, updated soft groupings pro-duce improved reflectance values, which in turn leads to more accurate comparisons of pixel similarity and better soft groupings. To further improve results, we make more complete use of BRDF information by partially groupingpixels that have similar reflectance except for albedo. Ona continuous surface, it is typical for reflectance to vary only in albedo, so we take advantage of this characteristic toprovide more data for estimation of non-albedo reflectance parameters. Under this scheme, we have been able to ob-tain reasonable reflectance estimates which have been ef- fectively used for relighting scenes.2. Reflectance estimationOur algorithm for reflectance estimation takes as input a single image, geometry that can be obtained by range scans or other means, and the light source position. Thereflectance models we use are a Lambertian model for dif-fuse reflectance and a single isotropic lobe from the Lafor-tune model 3 to represent other reflectance effects, suchas specular reflection. In terms of light direction L, surfacenormal N and viewing direction V , reflectance at a point x is formulated asI(x) = (x)N(x)L+c1(LV )+c2(N(x)L)(N(x)V )n (1)where the a