# Multi-view Stereo Beyond Lambert

 Hailin Jin hljin@cs.ucla.edu UCLA Stefano Soatto soatto@ucla.edu UCLA Anthony J. Yezzi ayezzi@ece.gatech.edu Georgia Institute of Technology
Project summary
In this project, we consider the problem of estimating the shape and radiance of an object from a calibrated set of views under the assumption that the reflectance of the object is non-Lambertian. While the reconstruction problem for Lambertian objects has been considered many times in the literature, few approaches have been proposed for non-Lambertian objects. Unlike traditional multi-view stereo, we do not seek for image-to-image correspondence, which is difficult to get due to the non-Lambertian nature. Instead, by studying geometric optics and radiometry, we propose a novel rank constraint on the radiance tensor field of the surface. We then exploit this constraint to define a discrepany measure between each image and the underlying model. Our approach automatically returns an estimate of the radiance of the object, along with its shape, represented by a dense surface. The former can be used to generate novel views that capture the non-Lambertian appearance of the object.
Related publications

In the heart of our algorithm lies the notation of radiance tensor field. Let $S$ be our surface. At each point $P \in S$, we construct the following matrix:

where $\rho_P(v_i,g_j)$ is the radiance. We show that the rank of this matrix is less than or equal to 2:

Therefore, we can express $R(P)$ as the sum of two rank 1 matrices.

Assuming a pinhole camera model, we can measure directly the radiance tensor at each point from images:

Our local discrepancy measure is nothing but a matrix norm for the difference between the estimated radiance tensor and modeled radiance tensor. We use the Frobenius norm as the matrix norm.

Integrating the local discrepancy measure over the whole surface, we obtain a cost functional of the unknown surface. We then use gradient descent flows to minimize the cost functional. The gradient descent flow is implemented using level set methods (Osher and Sethian 1988).

Once the shape is estimated, as a byproduct, we also have the radiance tensor ($R(P)$ or equivalently $s_1$, $d_1$, $s_2$, and $d_2$) for all the points on the surface. Therefore, given a novel viewpoint $g'$, we can interpolate $s_1(g')$ and $s_2(g')$ from existing $s_1$ and $s_2$. Note that $d_1$ and $d_2$ does not depend on $g'$ and therefore do not need to be interpolated.

Experimental results
All you need to obtain such results is a set of calibrated images. Unfortunately, taking high-quality pictures and calibrating them are not easy tasks. There are software packages available for camera calibration. We have been using the one by Jean-Yves Bouguet, which is available at http://www.vision.caltech.edu/bouguetj/calib_doc. The data we used are courtesy of Jean-Yves Bouguet and Radek Grzeszczuk (Intel Research) and Daniel Wood (University of Washington). The buddha data we use are publicly available from the OpenLF project at Sourceforge.