Aaron Hertzmann Ken PerlinMedia Research Laboratory
Department of Computer Science
Courant Institute of Mathematical Sciences
New York University
Painting is a particularly beautiful and expressive medium. An skilled artist can eloquently portray appearance, mood, emotion, space, the play of light on surfaces. Brush strokes and paint create a uniquely compelling medium. Our work aims to combine the beauty and expressivity of natural media with the flexibility of computer graphics.
Creating painterly animation and interfaces is one of the most important challenges in this area. Many experimental animators create wonderful short painterly animations with traditional media; however, these animations require the skills of a master and years of effort. Digital painterly animation promises to allow animators to create feature length animation in traditional animation styles.
We create paintings and animation by processing images and video. The basic approach is paint a rough sketch with a big brush, and then refine it with a smaller brush. The key is that the small brush only refines the image where there are differences between the painting and the source image, so that areas without much detail are left alone. This produces a painting with brush stroke sizes that vary over the image according to the details of the image. Individual strokes are placed to follow contours in the images.
Styles of Painting
We can change the painting style by changing the parameters to the algorithm.
Gallery of paintings and styles
By processing video sequences, we can create painterly animation. However, if we simply process each frame independently, the result will flicker in an undesirable way. Instead, we borrow an idea from experimental paint-on-glass animation: each frame of the animation is created by painting over the last frame. Only parts of the video that are changing get painted. In addition, we can estimate optical flow to move brush strokes around to follow the geometry.
Interactive ``living painting.''
This painterly animation method is featured on jazz cellist Erik Friedlander's DVD/VHS recording "Skin," available on Siam Records.
Relaxation and Weight Images
We can make the painting style more precise by writing as the solution to an energy optimization problem and trying to decrease the energy as much as possible. Procedural stroke textures can be added for additional appeal.
We can give more control over the painterly style to the artist/animator through the use of weight images. The weight image allows the user to specify different styles to different parts of a painting. In the following example, we use a weight image to specify more emphasis for the characters in the scene and less emphasis in the background.
SIGGRAPH 2001 paper
on example-based rendering
NPAR 2000 paper on painterly video and interaction
SIGGRAPH 98 paper on still image processing
NYU Technical Report on painting by relaxation
Project page: Non-photorealistic
Project page: Image analogies
The HuangShan (mountain) image is used by kind permission of
Topaz images appear courtesy of Erik Friedlander and SIAM Records.
Stockholm photographs courtesy Philip Greenspun