first try in CGFX - tracking and compositing

Been a while since I posted - and for a good reason! That however doesnt necessarily mean that I haven't been busy cooking new content. On the contrary :)
So This month I am going to try and catch up with you folks, also document my adventures with cg software.

Today I want to look back at an assignment that was presented to me by university. Let me start by posting the whole thing:


http://www.vimeo.com/23987995

No, my inspiration for this does not come from Jurasic park or Jumanji. I'd say mostly "Blade runner" and a game from the 90ies called "Flashback"

The footage that I got was grainy, and with a weird resolution (960x540) Pal (25 fps). The brief was to composit some elements over said footage by using after effects and tracking. The footage was only 11 seconds and I managed to get it ready in 11 days.

Here is how...

Motion tracking with matchmover turned out to be a semi-success , even though it wasnt very precise. Since the camera is not really moving in 3d space, just panning up- matchmover had a hard time determining where exactly are the markers in xyz. With some tweaking and even minor brute force I managed to at least get a coordinate system to use in maya.

Bringing the tracking data to maya revealed that there was a minor twitch of the camera. I fixed that by dropping two new keyframes over the group that contained the reference 3d objects and also deleting some keyframes. Some might say that this shot might have done without 3d tracking. The thing is that I really wanted to get to know the software+ there is some perspective distortion (on the roof of the building and the top of the entrance) as the camera pans up.

The first step for me was getting the trackers to work, in order to be able to pin things to the footage. Seeing if things would flow with the 2d and 3d tracking was also the right time to plan the shot. So After I got my 3d tracking done, I made a bunch of really basic non textured models and animated some of them. Later they were replaced by proper ones.

2d tracking with mocha

Just had to use mocha tracking, in order to pin some texture to the walls of the building. It turned out to be one intuitive and useful piece of software. Again- tutorials from youtube :P

I discovered that graininess and lack of focus on the background elelments make the tracking markers unstable. Thankfully that was something that could be fixed quickly in Mocha. It even has an "auto" button. Adjusting trackers with a mouse can be paintstaking , because one has to point at an exact pixel of the screen. One cool thing that I found out is that using the laptop's touchpad for this sorta thing is actually better than the mouse. By some lucky coincidence my mouse broke down at that time and forced me to do so.
Mocha was also excellent for rotoscoping (export shape data). Its capable of pinning the nodes of a mask to trackers, dynamically masking a specific area (used that for the top of the roof).



I pasted that tracking data onto a bunch of solids in aftereffects and then replaced them with testing uv grids. That is how I made a UV map of the faces of the building. Making its layer semi-transparent so it was possible to see both uv space and the building details underneath it.



After rendering that out of after effects and bringing it to photoshop, I laid it over the 960x540 original uv map image - using the free transform tool to unwrap it. Knowing where exactly the windows are enabled me to paint detail on the building walls. Its all hand painted in photoshop with the help of my cheap tablet.



The sky color mask was created with the help of the new amazing roto brush in after effects. I dropped a green tint on it, to make it look more surreal and to fit the mood of the shot I was aiming for.

Compositing the cgi footage:

In order to get my render to look right, I first made sure that I am happy with the colors in after effects and the building detail that I added has the right colors too. Then I rendered a single frame (Aftereffects) out of that and brought it as a frame in the tracking image sequence( that MAYA is using for the tracking camera ). I also opened it in photoshop, so I can pick the colors while painting textures on the cgi elements.
Uv work and texturing was all done in blender..

Scrubbing to the aftereffects touched frame in Maya, I was able to set up the light and materials as close as possible to blend in. A trick I did to make the cgi blend in more was using some reflectivity on the metalic materials and a sky dome mesh (which was set not to render, but to be visible in reflections). There is also a surface shader on the faces that I needed to key out in after effects. I separated the car in the sky from the landing car to two separate render layers- so it will be easy to quickly blur out the first in after effects- creating the illusion of depth of field and improving the composition of the shot.





Bringing the 3d footage to aftereffects revealed a number of issues that needed my attention. Some of it was covering the entrance of the building. I fixed that by getting some mocha shape data to aftereffects for the front of the entrance. I dropped some gaussian blur filthers, outline blur and hue/saturation on the elements. Some things needed to be desaturased. The cgi stuff was still too crisp so one last thing i did was dropping a slight Noise filther on it.

Things that can be improved:
The motion tracking data can be a lot better. Time restrains really left their mark and there are some minor issues that need attention. When the camera pans up, the top of the building shape twitches a bit, revealing a slight blue border around the bars of the roof top. The piston on the side of the landing zone is dropping a slightly unrealistic shadow which shouldnt normally be there considering that the light source is not close to it- its day light which should cast softer shadows.

All and all, this excercise was a really challenging and thus also fun one for me. I hope that I get another chance to do this sort of work again. Hopefully not with after effects! After effects makes it hard to find problems when problems occur. The layering way of approaching compositing is also a bit limiting. I had to toggle layers on and off to see where the problem is. That is in my oppinion a very inneficient way to deal with compositing, especially when there are more than ten layers in the shot. Nuke, Shake,autodesk composit (formerly Toxic) are all excellent packages to do cgi compositing with a proper node-based approach. The problem is that Universities (or at least mine) are not teaching it at the time that I'm writing this. :)