Multi-view Stereo Beyond Lambert
|
|||||||||
| Project summary | |||||||||
| In this project, we consider the problem of estimating the shape and radiance of an object from a calibrated set of views under the assumption that the reflectance of the object is non-Lambertian. While the reconstruction problem for Lambertian objects has been considered many times in the literature, few approaches have been proposed for non-Lambertian objects. Unlike traditional multi-view stereo, we do not seek for image-to-image correspondence, which is difficult to get due to the non-Lambertian nature. Instead, by studying geometric optics and radiometry, we propose a novel rank constraint on the radiance tensor field of the surface. We then exploit this constraint to define a discrepany measure between each image and the underlying model. Our approach automatically returns an estimate of the radiance of the object, along with its shape, represented by a dense surface. The former can be used to generate novel views that capture the non-Lambertian appearance of the object. | |||||||||
| Related publications | |||||||||
|
|||||||||
| Radiance tensor | |||||||||
In the heart of our algorithm lies the notation of radiance tensor
field. Let $S$ be our surface. At each point $P \in S$, we construct the
following matrix: where $\rho_P(v_i,g_j)$ is the radiance. We show that the rank of this matrix is less than or equal to 2:  Therefore, we can express $R(P)$ as the sum of two rank 1 matrices.  Assuming a pinhole camera model, we can measure directly the radiance tensor at each point from images:  Our local discrepancy measure is nothing but a matrix norm for the
difference between the estimated radiance tensor and modeled radiance
tensor. We use the Frobenius norm as the matrix norm. |
|||||||||
| Shape and radiance estimation | |||||||||
|
Integrating the local discrepancy measure over the whole surface, we
obtain a cost functional of the unknown surface. We then use gradient
descent flows to minimize the cost functional. The gradient descent flow
is implemented using level set methods (Osher and Sethian 1988).
Once the shape is estimated, as a byproduct, we also have the radiance tensor ($R(P)$ or equivalently $s_1$, $d_1$, $s_2$, and $d_2$) for all the points on the surface. Therefore, given a novel viewpoint $g'$, we can interpolate $s_1(g')$ and $s_2(g')$ from existing $s_1$ and $s_2$. Note that $d_1$ and $d_2$ does not depend on $g'$ and therefore do not need to be interpolated. |
|||||||||
| Experimental results | |||||||||
|
All you need to obtain such results is a set of calibrated
images. Unfortunately, taking high-quality pictures and calibrating them
are not easy tasks. There are software packages available for camera
calibration. We have been using the one by Jean-Yves
Bouguet, which is available at http://www.vision.caltech.edu/bouguetj/calib_doc.
The data we used are
courtesy of Jean-Yves Bouguet and Radek Grzeszczuk (Intel
Research) and Daniel Wood (University of Washington).
The buddha data we use are publicly available from
the OpenLF project at
Sourceforge.
input, shape, radiance shape evolution, radiance movie input, shape, radiance shape evolution, radiance movie |
|||||||||
| Sponsors | |||||||||
|
This project is supported by NSF grants IIS-0208197 and CCR-0133736, ONR
grant N00014-02-1-0720, AFOSR grant F49620-03-1-0095, and Intel grant 8029.
© 2003 Hailin Jin, Stefano Soatto and Anthony J. Yezzi. Please send your comments to hljin@cs.ucla.edu. Last updated on January 8, 2004. BACK to Hailin Jin's homepage |
