Gianfranco Doretto / Research / Project
Dynamic texture segmentation
A variational approach for spatio-temporal segmentation
Description
By looking at a scene, humans are able to localize and identify
different dynamic visual processes, for example, we can localize and
identify stem from smoke or haze. Given a sequence of images of a
moving scene, we would like to analyze the scene and decompose the
image plane into regions where different dynamic visual processes take
place.
The problem of decomposing the image plane into regions can be
addressed by classifying portions of the image into a number of
classes, for instance grass, dirt, smoke, bushes or water. For the
most part, such a classification can be accomplished successfully by
looking at simple image statistics, such as color or
intensity. However, in many situations these are not sufficient, and
therefore it may be beneficial to look at spatio-temporal
statistics, and attempt to classify different portions of the scene
based not on the statistics of one single image, but on how the
statistics of an image change over time during a sequence.
Modeling the (global) spatio-temporal statistics of the entire image
can be a daunting task due to the complexity of natural scenes. An
alternative consists of choosing a simple class of
models, and simultaneously estimating regions and their model
parameters in such a way that the data in each region is optimally
modeled by the estimated parameters. This naturally results in a
segmentation problem.
In our approach we make the assumption that the regions with different
dynamic visual processes have boundaries that either do not change or
change in a stationary fashion. Within each region we model the
spatio-temporal statistics with simple linear-Gaussian models, and
infer model parameters as well as region boundaries in a
variational optimization framework. Our technique is effective in
segmenting regions that differ in their dynamics even when spatial
statistics are identical, and it can be useful in outdoor autonomous
navigation.
The main contributions of our approach are:
- A formalization of the problem of segmenting dynamic visual processes by jointly exploiting their spatial and temporal statistics.
- A representation of the joint spatial and temporal statistics that uses dynamic texture models and system identification tools like subspace angles.
- A variational framework for segmentation which combines region-based low-levelcues with the higher level representation of the joint spatial and temporal statistics.
- Extension of the Mumford-Shah functional for spatio-temporal textures.
- Demonstration that our method is able to segment based on a jointly spatial and temporal statistics, based on only spatial statistics, and based on only temporal statistics, which means that our method is effective in segmenting regions that differ in their dynamics even when spatial statistics are identical.
Results
Smoke on the water
In this first experiment the two dynamic textures (smoke and ocean
waves) are very different both in the dynamics and in the
appearance. The boundaries of the two dynamic textures are well
defined, and our algorithm works by exploiting both spatial and
temporal cues. Note that we are not segmenting a point process, but
rather statistical distributions in space and time. Therefore, there
is a minimum region of integration in order to capture such
statistics. This is why the estimated boundaries do not perfectly
match the contour that separates smoke from ocean. The same
observation holds for the next experiments. (Click on the images to
view them in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [571Kb]
Watch the evolution of the segmenting contour.
Download .avi movie [571Kb]
|
|
|
|
Segmentation by changing texture
In this experiment we segment two dynamic textures that differ only
for the texture orientation, but that share the same dynamics and
general appearance (grayscale values). Here our algorithm works by
exploiting only spatial cues since temporal ones remain the same over
all the image plane. (Click on the images to view them in full
size.)
Watch the evolution of the segmenting contour.
Download .avi movie [453Kb]
Watch the evolution of the segmenting contour.
Download .avi movie [453Kb]
|
|
|
|
Segmentation by changing dynamics
In this experiment we segment two dynamic textures that are identical
in appearance, but differ in the dynamics. Note that this particular
segmentation problem is quite difficult, even for human
observers. Segmentation is obtained exclusively on the basis of the
temporal properties of the dynamic texture. This demonstrates one
important novelty of our approach. (Click on the images to view them
in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [443Kb]
Watch the evolution of the segmenting contour.
Download .avi movie [443Kb]
|
|
|
|
... and fire in the sky
In this experiment we segment a challenging sequence, since the
regions where the dynamic textures are defined (in particular, the
flame texture), are also changing in time. The segmentation returns a
visually acceptable result, which shows the robustness of our
approach. (Click on the images to view them in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [1265Kb]
Watch the evolution of the segmenting contour.
Download .avi movie [1265Kb]
|
|
|
|
Related publications
- Doretto, G., Cremers, D., Favaro, P., and Soatto, S.
Dynamic texture segmentation. In Proceedings of IEEE International Conference on Computer Vision, pp. 1236–1242, Nice, France, October 2003.
Details BibTeX PDF (241.2kB ) - Saisan, P., Doretto, G., Wu, Y. N., and Soatto, S.
Dynamic texture recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 58–63, Kauai, Hawaii, USA, December 2001.
Details BibTeX PDF (315.7kB ) - Doretto, G., Chiuso, A., Wu, Y. N., and Soatto, S.
Dynamic textures. International Journal of Computer Vision, 51(2):91–109, 2003.
Details BibTeX PDF (2.6MB )