Gianfranco Doretto / Research / Project

Dynamic texture segmentation

A variational approach for spatio-temporal segmentation

Description

By looking at a scene, humans are able to localize and identify different dynamic visual processes, for example, we can localize and identify stem from smoke or haze. Given a sequence of images of a moving scene, we would like to analyze the scene and decompose the image plane into regions where different dynamic visual processes take place.
The problem of decomposing the image plane into regions can be addressed by classifying portions of the image into a number of classes, for instance grass, dirt, smoke, bushes or water. For the most part, such a classification can be accomplished successfully by looking at simple image statistics, such as color or intensity. However, in many situations these are not sufficient, and therefore it may be beneficial to look at spatio-temporal statistics, and attempt to classify different portions of the scene based not on the statistics of one single image, but on how the statistics of an image change over time during a sequence.
Modeling the (global) spatio-temporal statistics of the entire image can be a daunting task due to the complexity of natural scenes. An alternative consists of choosing a simple class of models, and simultaneously estimating regions and their model parameters in such a way that the data in each region is optimally modeled by the estimated parameters. This naturally results in a segmentation problem.
In our approach we make the assumption that the regions with different dynamic visual processes have boundaries that either do not change or change in a stationary fashion. Within each region we model the spatio-temporal statistics with simple linear-Gaussian models, and infer model parameters as well as region  boundaries in a variational optimization framework. Our technique is effective in segmenting regions that differ in their dynamics even when spatial statistics are identical, and it can be useful in outdoor autonomous navigation.
The main contributions of our approach are:
  • A formalization of the problem of segmenting dynamic visual processes by jointly exploiting their spatial and temporal statistics.
  • A representation of the joint spatial and temporal statistics that uses dynamic texture models and system identification tools like subspace angles.
  • A variational framework for segmentation which combines region-based low-levelcues with the higher level representation of the joint spatial and temporal statistics.
  • Extension of the Mumford-Shah functional for spatio-temporal textures.
  • Demonstration that our method is able to segment based on a jointly spatial and temporal statistics, based on only spatial statistics, and based on only temporal statistics, which means that our method is effective in segmenting regions that differ in their dynamics even when spatial statistics are identical.

Results

Smoke on the water

In this first experiment the two dynamic textures (smoke and ocean waves) are very different both in the dynamics and in the appearance. The boundaries of the two dynamic textures are well defined, and our algorithm works by exploiting both spatial and temporal cues. Note that we are not segmenting a point process, but rather statistical distributions in space and time. Therefore, there is a minimum region of integration in order to capture such statistics. This is why the estimated boundaries do not perfectly match the contour that separates smoke from ocean. The same observation holds for the next experiments. (Click on the images to view them in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [571Kb]

Segmentation by changing texture

In this experiment we segment two dynamic textures that differ only for the texture orientation, but that share the same dynamics and general appearance (grayscale values). Here our algorithm works by exploiting only spatial cues since temporal ones remain the same over all the image plane. (Click on the images to view them in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [453Kb]

Segmentation by changing dynamics

In this experiment we segment two dynamic textures that are identical in appearance, but differ in the dynamics. Note that this particular segmentation problem is quite difficult, even for human observers. Segmentation is obtained exclusively on the basis of the temporal properties of the dynamic texture. This demonstrates one important novelty of our approach. (Click on the images to view them in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [443Kb]

... and fire in the sky

In this experiment we segment a challenging sequence, since the regions where the dynamic textures are defined (in particular, the flame texture), are also changing in time. The segmentation returns a visually acceptable result, which shows the robustness of our approach. (Click on the images to view them in full size.)
Watch the evolution of the segmenting contour.
Download .avi movie [1265Kb]

Related publications

  • Doretto, G., Cremers, D., Favaro, P., and Soatto, S.
    Dynamic texture segmentation.
    In Proceedings of IEEE International Conference on Computer Vision, pp. 1236–1242, Nice, France, October 2003.
    Details   BibTeX   PDF (241.2kB )  
  • Saisan, P., Doretto, G., Wu, Y. N., and Soatto, S.
    Dynamic texture recognition.
    In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 58–63, Kauai, Hawaii, USA, December 2001.
    Details   BibTeX   PDF (315.7kB )  
  • Doretto, G., Chiuso, A., Wu, Y. N., and Soatto, S.
    Dynamic textures.
    International Journal of Computer Vision, 51(2):91–109, 2003.
    Details   BibTeX   PDF (2.6MB )