Gianfranco Doretto / Research / Project
Dynamic Texture Editing
Editing dynamic textures by changing model parameters
Description
The goal of this project is to design algorithms for synthesizing and
editing realistic sequences of images of dynamic scenes that exhibit
some form of temporal regularity. Such scenes include flowing water,
steam, smoke, flames, foliage of trees in wind, crowds, dense traffic
flow etc.
Algorithms that aims at synthesizing and editing video sequences
traditionally use a physical model of the scene. This approach is
commonly known as physics-based rendering (PBR). While this approach
is very principled and it allows full editing power of the scene, on
the other hand it is usually computationally intensive, targeted to
specific scenes, to not mention that in general it is difficult to
build physical models that well explain certain phenomena, and for
certain applications these approaches may be overkill. These reasons
motivate the image-based rendering approaches (IBR) that try to use
images of real scenes to synthesize other images. IBR techniques
typically allow to get impressive results without much computation,
and are often simpler to code. One major drawback of IBR techniques is
often their lack of flexibility in terms of editing power.
We use an IBR approach that relies on a simple statistical model for
the video sequences, and study to what extent model parameters can be
modified to provide as much editing power as possible. Despite the
difficulty of editing IBR models, we found that our approach allows to
interactively change and reverse the speed of the dynamic visual
process of the scene, change its spatial frequency content, an change
the statistical source that generates the simulation. The editing
parameters allow to synthesize several interesting visual effects, and
every kind of manipulation can be carried out online and in
real-time.
Notice that this framework, combined with recognition, could be used
to infer higher level properties of the observed dynamic visual
process. For example, one could estimate the energy of the wind in a
certain natural environment, or the wave energy flow in ocean waves.
The main contributions of our approach are:
- A simple and efficient IBR framework for modifying the temporal and spatial behavior of dynamic textures.
- The definition of the conditions under which model parameters of a dynamic texture can be modified.
- Thorough explanation of the relationship between model parameters and visual appearance of the scene.
- Several demonstrations of the power of this approach, that allow online and real-time interaction with the editor.
Results
The following examples demonstrate the power of our approach to
extrapolate and manipulate new video sequences. Given a training
sequence we apply the learning procedure for dynamic textures, and
extract the parameters of the model. We then simulate and edit the
model to synthesize new video sequences.
The images of the movies show on the left a depiction of the scene,
and on the right the corresponding values of some editing
parameters. The first slider depicts the speed, that ranges from 0 to
3, or from −3 to 0, times the speed of the training
sequence. The second slider represents the intensity of the driving
noise of the simulation; it ranges from 0 to 3 times the intensity of
the driving noise of the training sequence. The last three sliders
depict the weights that determine the spatial frequency content of the
video sequence. When they are set to one the simulation has the same
frequency content of the training sequence. The first slider weights
the coarse scale frequencies, the second the middle scale frequencies,
and the last one the fine scale frequencies.
Note that the following examples include only a selected subset among
all the possible changes of the editing parameters. For example, one
could include rotations of the spatial frequencies and the state
space, or process the color channels independently, or perform some
other operations.
Note that the learning procedure has been applied directly to the raw
data, and no preprocessing has been performed. Also, for portability
issues, the .avi movies are MPEG compressed (video coder V1), and the
quality of the synthesized images has degraded accordingly.
Smoke
In the following example one can see that increasing the intensity of
the driving input results in an apparently more "turbulent"
smoke, or that amplifying the coarse frequency components results in a
thinner "hazy" smoke. Also, one can make the smoke appear more
"grainy" or "patchy," and can adjust or reverse
the speed at his own wish.
Download .avi movie [609Kb]
Download .avi movie [609Kb]
Ocean waves
In this example you can see how to produce a "rougher" sea
movement with larger waves by amplifying intensity and coarse and fine
scales, or how to produce a "lake effect" with more gentle
and smooth waves. Finally, increasing the intensity and the fine
scale, while decreasing the coarse and middle scale results in a
"rain effect", like rain pouring on a pond.
Download .avi movie [3.25Mb]
Download .avi movie [3.25Mb]
Fountain
This example shows how playing with the intensity and scale parameters
results in interesting effects that appear to be the results of
changing the nozzle of the fountain, from a "spurty" fountain,
to a "spraylike" fountain. Also, the fountain can be slowed down
and brought to a complete stop.
Download .avi movie [2.58Mb]
Download .avi movie [2.58Mb]
Fire
Here we show the effects of altering a dynamic texture of a flame,
including changing the spatial scales, speed, direction
etc. Non-realistic effects can also be achieved by altering the
dynamics of each color component independently.
Download .avi movie [2.63Mb]
Download .avi movie [2.63Mb]
Related Publications
- Doretto, G. and Soatto, S.
Editable dynamic textures. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 137–142, Madison, Wisconsin, USA, June 2003.
Details BibTeX PDF (507.5kB ) - Doretto, G. and Soatto, S.
Editable dynamic textures. In Conference Abstracts and Applications of SIGGRAPH '02, pp. 177, San Antonio, Texas, USA, July 2002.
Details BibTeX PDF (822.3kB ) - Doretto, G. and Soatto, S.
Editable dynamic textures. Technical Report TR020001, UCLA Computer Science Department, 2002.
Details BibTeX PDF (832.5kB ) - Doretto, G., Chiuso, A., Wu, Y. N., and Soatto, S.
Dynamic textures. International Journal of Computer Vision, 51(2):91–109, 2003.
Details BibTeX PDF (2.6MB )