Multiview Stereo Beyond Lambert


Project summary  
In this project, we consider the problem of estimating the shape and radiance of an object from a calibrated set of views under the assumption that the reflectance of the object is nonLambertian. While the reconstruction problem for Lambertian objects has been considered many times in the literature, few approaches have been proposed for nonLambertian objects. Unlike traditional multiview stereo, we do not seek for imagetoimage correspondence, which is difficult to get due to the nonLambertian nature. Instead, by studying geometric optics and radiometry, we propose a novel rank constraint on the radiance tensor field of the surface. We then exploit this constraint to define a discrepany measure between each image and the underlying model. Our approach automatically returns an estimate of the radiance of the object, along with its shape, represented by a dense surface. The former can be used to generate novel views that capture the nonLambertian appearance of the object.  
Related publications  


Radiance tensor  
In the heart of our algorithm lies the notation of radiance tensor
field. Let $S$ be our surface. At each point $P \in S$, we construct the
following matrix: where $\rho_P(v_i,g_j)$ is the radiance. We show that the rank of this matrix is less than or equal to 2: Therefore, we can express $R(P)$ as the sum of two rank 1 matrices. Assuming a pinhole camera model, we can measure directly the radiance tensor at each point from images: Our local discrepancy measure is nothing but a matrix norm for the
difference between the estimated radiance tensor and modeled radiance
tensor. We use the Frobenius norm as the matrix norm. 

Shape and radiance estimation  
Integrating the local discrepancy measure over the whole surface, we
obtain a cost functional of the unknown surface. We then use gradient
descent flows to minimize the cost functional. The gradient descent flow
is implemented using level set methods (Osher and Sethian 1988).
Once the shape is estimated, as a byproduct, we also have the radiance tensor ($R(P)$ or equivalently $s_1$, $d_1$, $s_2$, and $d_2$) for all the points on the surface. Therefore, given a novel viewpoint $g'$, we can interpolate $s_1(g')$ and $s_2(g')$ from existing $s_1$ and $s_2$. Note that $d_1$ and $d_2$ does not depend on $g'$ and therefore do not need to be interpolated. 

Experimental results  
All you need to obtain such results is a set of calibrated
images. Unfortunately, taking highquality pictures and calibrating them
are not easy tasks. There are software packages available for camera
calibration. We have been using the one by JeanYves
Bouguet, which is available at http://www.vision.caltech.edu/bouguetj/calib_doc.
The data we used are
courtesy of JeanYves Bouguet and Radek Grzeszczuk (Intel
Research) and Daniel Wood (University of Washington).
The buddha data we use are publicly available from
the OpenLF project at
Sourceforge.
input, shape, radiance shape evolution, radiance movie input, shape, radiance shape evolution, radiance movie 

Sponsors  
This project is supported by NSF grants IIS0208197 and CCR0133736, ONR
grant N000140210720, AFOSR grant F496200310095, and Intel grant 8029.
© 2003 Hailin Jin, Stefano Soatto and Anthony J. Yezzi. Please send your comments to hljin@cs.ucla.edu. Last updated on January 8, 2004. BACK to Hailin Jin's homepage 