Some cartoon shading research

Why the long interest in cartoons?

A quick and cheap real time material, capable of pulling off a "cartoony" look would be useful to quickly show off models. It makes sense to apply a cartoon style rendering to a film/game that has cartoon style character designs to begin with. This style is still a niche that some titles have successfully exploited in order to stand out visually in the 3d crowd. It's a style that is quite popular on mobile games!

+ Cuts production time?
+  Cheap to render -even real time- can be used in both film and video games
+ Easy?
- Obviously fake
- Can look cheap if not done well. Technology alone doesn't necessarily pull of the great end look.

Lets look into some of the ingredients that can be used to achieve this look

Cell/soft Shading  
-We need to tweak or remove the falloff of a realistic shading to simulate the effect of cell shading. We can do this in a number of ways. Starting with a flat surface shader ("shadeless" in blender) will remove any realistic shading on the object.

By baking the shadows onto the texture- This seems to be a common method with low end games. Baking shadows on to texture is a common practice by AAA games too. They do that in a way that the player doesn't really notice it, as the technique (also known as Lightmap baking) is applied only to non-dynamic shadows that in theory don't need to change. Lightmaps are used to generate ambient occlusion maps, fake colour bleeding from light sources and bounce lights (global ilumination). Developers do that and many other things to optimise their levels, to squeeze more fps.

I made a kimba cartoon 3d fanart model to try this idea.

For the first time I did my 512x512 texture in inkscape. There are some great advantages in using vector software for cartoon styles- be it 2d or 3d. It's easy to tweak colours and curves without the need of a messy eraser,lasso-ing or millions of layers. You can also upscale your texture to any size you want. Its pixel precise and smooth. Every pixel counts.

For some cartoon character designs, it's better to use flat surface eyes (instead of spheres). I left space around the irises in the texture, making it possible to animate their movement by animating the UV.

But if I animate them, they will need to be on a separate uv map.

Depth map shadows can be useful to add to it

Depth map shadows are shadows cast by an object that interacts with a light source. They're cheap, but not completely realistic - lacking opacity/blur falloff calculation. These things can be procedurally faked with post effects.

I tend to avoid having these shadows as the default completely black. Its nice to give them some sort of a warm dark tone, possibly use a ramp, possibly set them to multiply and render them on their own render layer, so they are easy to tweak in post.

Matcaps
This is a very well known shading method to Zbrush users. Matcaps are a great way to get some material and light properties on your mesh- without the need to set up a light rig and tweak shader properties. Instead all that information is stored in a single image file. Map that file to the normals of your model and you get a sort of an overlay that fakes material properties and interaction with lights, based on the angle of the camera.

There is a number of ways one can create matcaps. I made three matcaps- the first one with inkscape, the second one with MaCrea and finaly the third one with blender's nodes.

Outlines can be generated by a matcap, but they don't work out as well as the other method. The matcap seems to be better suited for fake rim lights:

A rim light created by a matcap. As you can see coordinates are set to "normal", I toned down the color value (it acts as a sort of a layer transparency in this case) and set the material layer to "Multiply". This is how you get matcaps in Blender.

 --> 

To create the matcap for the rim, I took a soft outline matcap I made in Inkscape, inverted its value and set its layer to screen, so I can give that black color a bit of a warm brown hue. This avoids desaturation/ugly shadows.

+This is good when you want to quickly preview the mesh in a closer to rendering situation (zbrush).

+It can be used to produce a cartoon outline and at the same time things like a rim light, secularity and soft shading- its very easy to tweak those by editing a single matcap image file or combining multiple matcaps.

+Some video games have successfully used it to cheaply create unique visual styles.

-Looking at a matcapped model is like having your viewport camera be the lightsource.

It doesnt look as if you're rotating around an object, but more like the object rotating on a clay pottery table. This can throw off the audience if the camera is orbiting around the target.

Cartoon outlines (outlines)
Mesh clone + normals trick- This popular method ( almost sure that Okami and Zelda windwaker use it) works in most real time rendering viewports. I found it to be the easiest approach.
To get it, simply duplicate your mesh,  then give the copy any dark material (uv texture, matcap,solid black color,etc). Flip its normals. Scale all its faces along the normals (alt+s in blender) , so its slightly bigger than the mesh- this also controls the global line thickness.

Tweak the outile mesh with your sculpting tools. The grab and smooth brushes are surprisingly good at manipulating the amount of outline, depending on the angle..

You can do this same thing with the solidify modifier if you're too lazy ;)

--disadvantages: Dublicating the mesh doubles the polycount, hence processing power. It also introduces the complexity of linking it to the character mesh. This should be done procedurally to avoid headaches when dealing with blendshapes. Perhaps doing a weird mesh proxy trick? I'll let you know if I find out.

Cartoon lines and shading can also be achieved by using the z-buffer pass ,the normal pass, and other passes - run those through a node tree and create the effect in post.

There is a lot more written on the topic here

http://blendernpr.org/bnpr-exclusive-edge-nodes-part-1/ 

My biggest issue with that can be

- Lack of full control over specific lines/shadows

- Lack of interactive feedback when tweaking values- freestyle render engine for example still needs to render out an image to show the lines. It would be best if it could update and preview in the opengl display.