CS798 Assignment 1 : Painterly rendering

Introduction

Painterly rendering is a technique that automatically creates images that have handed-painted appearance from a photograph.

One such algorithm has been described by Aaron Hertzmann in "Painterly Rendering with Curved Brush Strokes of Multiple Sizes" , from SIGGRAPH'98 conference. The algorithm applies successive layers of strokes on the canvas. The radius of the strokes decreases to convey more details at each iteration. The strokes are applied where the canvas differ sufficiently from the reference image. The strokes follow splines curves of nearly constant color in the reference image.

For my extra, I implemented some prettier strokes, by adding some height and opacity map texture. This extension has been proposed by Aaron Hertzmann in "Fast Paint Texture", for the NPAR 2002 conference.

Part I : Painterly Rendering with Curved Brush Strokes of Multiples Sizes

Implementation Details

The program is written in Python for the flexibility of the interface and uses C++ with OpenGL to generate efficiently the geometry of the strokes of the final painting. The Python Imaging Library allows to load and save images easily. I use PIL_usm to apply a Gaussian blurring to the reference image. I use the code provided by Craig Kaplan, painthelp.cpp and paint.py (without libart) and also some parts of color.h, Point.h, Stroke.h, and Stroke.cpp, donated to the course by Aaron Hertzmann.

Parameters

The paper defines a set of parameters that help to produce the different painting style. These parameters are constants in my program.

Approximation threshold T: error between reference image and canvas that is allowed.
Brush sizes R(1..n) : list of brush sizes that conveys the level of details in each layer iteration.
Curvature filter f_c : constraint strokes curvature.
Blur factor f_s : size of bluring.
Stroke length [min..max].
Opacity alpha : alpha value of the color of all the strokes.
Grid size f_g: spacing of brush strokes.
Color jitter jr, jg, jb : red, green, blue values randomly added to stroke color sometimes (-3...+3 over 255).

Parameters	Impressionist	Expressionist	Colorist Wash	Pointillist
Threshold	100	50	200	100
Brush sizes	8, 4, 2	8, 4, 2	8, 4, 2	4, 2
Curvature filter	1	0.25	1	1
Blur factor	0.5	0.5	0.5	0.5
Stroke length	[ 4 .. 16 ]	[ 10 .. 16 ]	[ 4 .. 16 ]	[ 1 ]
Alpha	1.0	0.7	0.5	1.0
Grid size	1.0	1.0	1.0	0.5
Color jitter	None	None	[-3..+3]	[-3..+3]

Note :
No particular interface is implemented.
Jitter for hue, jh, saturation js and value jv not used.

Algorithm Steps

repeat for each brush radius Ri :

Blur reference image by f_s * Ri.
Compute the blur reference image gradient for each point using the Sobel filtered luminance.
Make splines strokes that have control points that roughly match (corresponding to threshold the isocontours of the reference image.
Render randomly all the strokes of radius Ri (tesselation as triangle strip along cubic B-spline of stroke control point).

Explanation in images

Results I : Different Brush Layers

Results II : Different Painting Styles

Part II : Fast Paint Texture

Fast Paint Texture is a technique for simulating the physical appearance of paint strokes under lighting. The list of strokes generated by my painterly rendering program produces an height field image by rendering the brush strokes with texture height map assigned to each stroke.
To implement Fast Paint Texture using the potential of OpenGL I modify my code that renders the strokes, to use OpenGL instead of libart. The given Stroke.cpp class provided by Aaron Hertzmann helped me a lot. I used the following opacity and height map for the strokes.

Stroke Height Map
Stroke Opacity Map

Notes :

I modified the stroke height map given in the paper. I darken it (0 .. 0.1) since I add a constant to the color of the texture when more strokes are painted. Since black represents 0 no height, by the last iteration the texture needs to be whiter since there are many layers of stroke underneath.
I also modified the opacity map given in the paper. I smooth the gradient change.

Explanations

I initially create a RGBA texture using the loaded stroke height and opacity maps.
The RGBA texture is mapped to the stroke tesselation, when strokes are determined.
When I render the final height field I use alpha blending and I also add to the color of the RGBA texture, a color (starting from black) that increasly get to white, using the glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_ADD) multi-texture feature of OpenGL.
I use Gimp to bump map, the rendered painting with the height field produced by my program. I turn on the compensate for darkeness option in order to simulate better the lighting. I could have used my ray-tracer to produce the bump mapping but I thought Gimp is doing it better and I may be a tiny lazy....

Results

Comments

The effect of texture differs according to the painting style (i.e. impressionist, expressionist, colorist wash or pointillist) of the painterly rendered picture. The stroke height map that I used, may be more appropriate in some painting style, like impressionist and expressionist. The result is not deceiving in all cases, but for example colorist wash is often a unique layer of wet media and applying bump mapping to it, produces thicker and dryer strokes that may be desired. On the other hand, texturing the strokes is ideal in oil looking painting, like the ones rendered with impressionist or expressionist styles.

Conclusion

I really enjoyed doing the painterly rendering program and experimenting it on different images. I found that my extra to add texture to the strokes produces a nice effect that is noticeable and visually interesting. I had trouble at first to figure out the easy and fast implementation the "Fast Paint Texture" talks about, but Kevin Moule pointed me out the right function of OpenGL to use.

References

Hertzmann, A. 1998. Painterly Rendering with Curved Brush Strokes of Multiple Sizes. In SIGGRAPH 98 Conference Proceeding, p453-460.
Hertzmann, A. 2002. Fast Paint Texture. In NPAR 2002 Conference.
Multitexturing with OpenGL