Multi-view Stereo Beyond Lambert

Hailin Jin
Stefano Soatto
Anthony J. Yezzi
Georgia Institute of Technology
Input image
Estimated shape
Estimated radiance
Project summary
In this project, we consider the problem of estimating the shape and radiance of an object from a calibrated set of views under the assumption that the reflectance of the object is non-Lambertian. While the reconstruction problem for Lambertian objects has been considered many times in the literature, few approaches have been proposed for non-Lambertian objects. Unlike traditional multi-view stereo, we do not seek for image-to-image correspondence, which is difficult to get due to the non-Lambertian nature. Instead, by studying geometric optics and radiometry, we propose a novel rank constraint on the radiance tensor field of the surface. We then exploit this constraint to define a discrepany measure between each image and the underlying model. Our approach automatically returns an estimate of the radiance of the object, along with its shape, represented by a dense surface. The former can be used to generate novel views that capture the non-Lambertian appearance of the object.
Related publications
| Radiance tensor | Shape and radiance estimation | Experimental results | Sponsors |

Radiance tensor
In the heart of our algorithm lies the notation of radiance tensor field. Let $S$ be our surface. At each point $P \in S$, we construct the following matrix:
Radiance tensor
where $\rho_P(v_i,g_j)$ is the radiance. We show that the rank of this matrix is less than or equal to 2:
Rank constraint
Therefore, we can express $R(P)$ as the sum of two rank 1 matrices.
Radiance tensor decomposition
Assuming a pinhole camera model, we can measure directly the radiance tensor at each point from images:
Measured radiance tensor

Our local discrepancy measure is nothing but a matrix norm for the difference between the estimated radiance tensor and modeled radiance tensor. We use the Frobenius norm as the matrix norm.
Local discrepany measure

Shape and radiance estimation
Integrating the local discrepancy measure over the whole surface, we obtain a cost functional of the unknown surface. We then use gradient descent flows to minimize the cost functional. The gradient descent flow is implemented using level set methods (Osher and Sethian 1988).

Once the shape is estimated, as a byproduct, we also have the radiance tensor ($R(P)$ or equivalently $s_1$, $d_1$, $s_2$, and $d_2$) for all the points on the surface. Therefore, given a novel viewpoint $g'$, we can interpolate $s_1(g')$ and $s_2(g')$ from existing $s_1$ and $s_2$. Note that $d_1$ and $d_2$ does not depend on $g'$ and therefore do not need to be interpolated.

Experimental results
All you need to obtain such results is a set of calibrated images. Unfortunately, taking high-quality pictures and calibrating them are not easy tasks. There are software packages available for camera calibration. We have been using the one by Jean-Yves Bouguet, which is available at The data we used are courtesy of Jean-Yves Bouguet and Radek Grzeszczuk (Intel Research) and Daniel Wood (University of Washington). The buddha data we use are publicly available from the OpenLF project at Sourceforge.

input, shape, radiance shape evolution, radiance movie

input, shape, radiance shape evolution, radiance movie

This project is supported by NSF grants IIS-0208197 and CCR-0133736, ONR grant N00014-02-1-0720, AFOSR grant F49620-03-1-0095, and Intel grant 8029.
2003 Hailin Jin, Stefano Soatto and Anthony J. Yezzi.
Please send your comments to
Last updated on January 8, 2004.

BACK to Hailin Jin's homepage