painterly rendering

Aaron Hertzmann      Ken Perlin

Media Research Laboratory
Department of Computer Science
Courant Institute of Mathematical Sciences
New York University

Painting is a particularly beautiful and expressive medium. An skilled artist can eloquently portray appearance, mood, emotion, space, the play of light on surfaces. Brush strokes and paint create a uniquely compelling medium. Our work aims to combine the beauty and expressivity of natural media with the flexibility of computer graphics.

Creating painterly animation and interfaces is one of the most important challenges in this area. Many experimental animators create wonderful short painterly animations with traditional media; however, these animations require the skills of a master and years of effort. Digital painterly animation promises to allow animators to create feature length animation in traditional animation styles.

Our approach

Coarse-to-fine painting

We create paintings and animation by processing images and video. The basic approach is paint a rough sketch with a big brush, and then refine it with a smaller brush. The key is that the small brush only refines the image where there are differences between the painting and the source image, so that areas without much detail are left alone. This produces a painting with brush stroke sizes that vary over the image according to the details of the image. Individual strokes are placed to follow contours in the images.

Source image

Rough sketch	Intermediate sketch

Final painting

Styles of Painting

We can change the painting style by changing the parameters to the algorithm.

Source image

``Impressionist''	``Expressionist''
``Watercolor''	``Pointillist''

Gallery of paintings and styles
Style interpolation

Painterly Animation

By processing video sequences, we can create painterly animation. However, if we simply process each frame independently, the result will flicker in an undesirable way. Instead, we borrow an idea from experimental paint-on-glass animation: each frame of the animation is created by painting over the last frame. Only parts of the video that are changing get painted. In addition, we can estimate optical flow to move brush strokes around to follow the geometry.

Interactive ``living painting.''

This painterly animation method is featured on jazz cellist Erik Friedlander's DVD/VHS recording "Skin," available on Siam Records.

Relaxation and Weight Images

We can make the painting style more precise by writing as the solution to an energy optimization problem and trying to decrease the energy as much as possible. Procedural stroke textures can be added for additional appeal.

We can give more control over the painterly style to the artist/animator through the use of weight images. The weight image allows the user to specify different styles to different parts of a painting. In the following example, we use a weight image to specify more emphasis for the characters in the scene and less emphasis in the background.

Source image

User-specified weight image	Final painting

Related Links

SIGGRAPH 2001 paper on example-based rendering
NPAR 2000 paper on painterly video and interaction
SIGGRAPH 98 paper on still image processing
NYU Technical Report on painting by relaxation

Project page: Non-photorealistic rendering
Project page: Image analogies

Credits

The HuangShan (mountain) image is used by kind permission of CND, Inc.
Topaz images appear courtesy of Erik Friedlander and SIAM Records.
Stockholm photographs courtesy Philip Greenspun