Advanced NPR Cell Shader ver 3 - and some thoughts on NPR lighting, and wishes for the BEER project

Ever since I released the Cell shader style material for Blender, there has been a rising number of people requesting from me to update it and continue developing it. I took some of your feedback and requests and managed to incorporate them in it. Admittedly there was also a huge break in which I kind of forgot about it as a whole, but some of you kept reminding me about it and I also kept remembering that I actually need to improve this material and add some features to it in order to continue production on SODA TRIP where the watercolory style is a must have.

So here it is! You can now download the cell shader - version 3 material from HERE:
Advanced Cell shader material version 3

Instructions how to use, some notes on new features and a wishlist to blender devs included in the video:

Advanced NPR Cell Shader 3 from todor imreorov on Vimeo.
The video contains some observations and ideas for b3d's npr renderer project- BEER.

New features:
- Watercolor effect!

- 2d screentones

- 2d gradients

- Receive shadows from external objects

- Textured mode and cells only mode

- Cell shading colorization

- Texture patches with multiply

- Spec bump pattern

-a much more modular and efficient node set up - under the hood.
The cell shader in completely modular - from version 2 and above! What's even better- it is much tidier. The matcap generator has become much tidier, easier to debug and extend. It is a part of the cell shader tools pipeline! This required a full redoing of node set ups from scratch.

Utility nodes (node groups) can be easily debugged, re-stacked and reused in different combinations and other shaders. These nodes are sort of like non destructive modifiers or effects.

Working on this cell shader allowed me to shape a number of concepts in my had as to how a potential NPR rendering engine would ideally work and what is currently missing in blender. You will hear some of these musings in the video anyway, but if you are too lazy to watch it- here is a summary:

Blender NPR BEER renderer ideas:

My wish for an ideal NPR shader is one that would allow me to use this sort of NPR effect modifiers on a per color-cell basis.

If freestyle has linesets, each of which has modifiers. A cell shader would have CELL sets, each of which would have cell shape/alpha modifiers (such as a watercolor effect, screentone effect and so on) and also cell color modifiers (such as 2d gradient, color patches, etc). 

Utility Nodes in version 3

Ideally a user would be able to set a cell set to receive shadows from isolated parts of itself and/or other objects. These geometry parts would be specifiable via a vertex group. So the user will be able to for example select only the nose to have a drop shadow, or only the chin to drop a shadow onto the neck. This is important for stylized shading. Just as important is also the ability to specify where you never want shadow to fall (example - the eyes) and where you never want  light to fall (example- inside the mouth). 

From there on it is kind of like the pipeline of my NPR cell shader material. Each cell goes through a number of modifiers that give the cell a specific style and flavor.

Once we get a greyscale render of all the cell sets, that information is then fed to the next stage of the process- THE COLOR STAGE.

In that stage we specify a texture or a vertex color layer as a base color. Then we can colorize the shading of each cell with a 2d gradient or a flat color- these things would ideally be keyframeable - so as to change them when the background is changed.

Additionally it would be nice to have the ability to have vertex based color patches that can also be keyframed, and also (tiling) texture patches. The option to operate on a cartoony character that is not UV unwrapped should be easily accessible.  

Then after all that is the Finessing stage.

In that stage you can add a rim light or a spec light- both of which can be made textury around the borders.

One of the design goals of this shader was to be flexible and to allow the artist to individually operate on elements. So it was important for me to separate cast shadows and normal based shadows - to be able to isolate and affect them with light and shadow masks; to be able to overlay color and texture patches, to affect the hue and saturation of cells; to be able to use 2d gradients and 2d screentones...

There are some other things on my wishlist, but these were and still are the big ones. There is more to be done to improve this shader, but there are also some annoying glsl bugs currently that trully discourage me from pushing it further to work on blender. 

Bugs encountered in blender:
However there are a couple of things in it that would break the glsl preview due to existing viewport limitations/bugs (as of 2.69)!

1. buffer shadows with "only shadow" option are pitch black in the glsl viewport. I filed a bug report for any developer interested in looking into it:
http://developer.blender.org/T37582

posted more about the limitation here:
http://blenderartists.org/forum/showthread.php?303682-Limitations-in-existing-Blender-features-you-would-like-to-see-resolved&p=2520945&viewfull=1#post2520945

2. For some reason using the screentones module (node group) or a 2d gradient (view/orco based vector) adds a white overlay on top of the material in glsl, but still works just fine as expected when rendering. It seems to be affecting the specular and rim light modules somehow.

3. Wishlist for Node groups in blender:
- A boolean switch (tick box) for 0/1 so we can have small switches instead of sliders when we want to enable/disable something.
- ability to expose drop down lists in the node group node- to select existing textures, vertex colors and other types of data. This would allow for a cleaner node material setup.