The first row in many rooms is now reserved for lecturers so I had to abandon one of my favorite seating places of past years. Further the exit of the main auditorium has been moved to its side, probably because people pushing their way out and people pushing their way in made the access to popular lectures a pain — literally. Further, the FMX had matured and grown and hence offered even more parallel lectures in the opposing building.

Shortly before ten I had my ticket, had set off my first tweet, got my welcome bag, had my ballpen at hand and took a seat in the second row: I was ready for this year’s conference! Since I was a bit early I came to see the last of the screened animations of the Filmakademie Baden-Württemberg, a satirical take on the juicy topic of Austrofashism in the 1930′s.

Going Mobile

Neil Trevett,
originally uploaded by Phil Strahl.

Then they rolled this year’s FMX trailer, in my opinion a fresh idea after last year’s rather featureless animation. Then the first speaker of the event got introduced, Neil Trevett from nVidia, incidentally nVidia being the event’s main sponsor. I was preparing myself for having some marketing mumbo-jumbo dropped onto me but it wasn’t that bad after all in his lecture Movie Making and More, All in the Palm of Your Hand.

First he lined out how proud nVidia is of their Quadro series, CUDA and that all of this year’s Oscar nominated VFX films used nVidia technology somewhere in the course of the production. Big whoop — Adobe can claim the same thing probably.

Then followed some more nVidia-green Power Point slides depicting how the market of the “classic”, stationary desktop computer was more or less saturated today, even the growth for notebooks and netbooks started to level out, yet the market for mobile devices such as smartphones and tablet-computers exploded exponentially, with the Android operating system already ahead of Apple’s iOS. And since nVidia wants to get a big slice of that cake as well they started development of the Tegra series, high-performance graphics processors with as little power consumption as possible.

But that’s not enough: The old x86 chip architecture is also in decline since tablets and smartphones don’t come with a constant connection to a power grid, so the old ARM architecture will become a big thing of the near future. For nVidia this is Project Denver, ARM with integrated GPUs, while Microsoft’s Windows 8 will also support this kind of hardware. nVidia also works together with Adobe to make the execution of Flash and AIR applications more economic on mobile devices. “Apart from playing back video, the main task of Flash seems to be blending and filtering. And we let the GPU do this.”

nVidia seems to be pretty tight with Motorola as well, because their cutting-edge processors chips are and will be inside cutting-edge Motorola hardware as a chart showed. And I realized that I will carry a laptop with me for a couple of years longer, because what I want and need to do on the go can’t be feasibly done on a tablet anytime soon. But for the average person who wants to listen to music, surf the web, read eBooks and write emails a tablet with a full-sized keyboard/dockingstation/battery-combo (like the ASUS Transformer) might be just what they need. USB-input-device support anyone? Ice Cream Sandwich?

In the very near future mobile devices will have quad-core processors hence multitasking should become as common as necessary, further the portable devices will provide autostereoscopic displays, such as the Nintendo 3DS. Last but not least the resolution with increase drastically to an estimated 2560 by 1600 pixels (nVidia boasts it “Extreme HD”) which effectively results in a resolution of 300 dpi on a 10-inch tablet — print quality.

But what about power consumption? “As paradox as it may sound but more cores save you more power: The excessive cores only turn on when an application needs them” Neil explained.

Gee thanks, Neil, that’s pretty interesting, consumer-wise, but what about our industry? “Tablets in the movie industry can be and currently are used in three different ways,” he continued. One possibility is to port software to mobile devices. I snorted mentally: Anybody who tried the incredibly limited version of Photoshop for Android knows that a smartphone is still way too underpowered to get real work done. “By 2014 mobile devices will have 100 times the performance of today.” So will CUDA also work on phones? “No, CUDA’s power consumption is one of the biggest stepping stones in that respect” Neil shrugged. But that didn’t matter to me, because “the next generation of tablets and smartphones will be aimed at the demands of artists and the creative, like by additionally offering stylus operability and pressure sensitivity. The first devices will probably be released this year.” I dumbly smiled at the poor person sitting to my right. I am so buying into this!

The next useful integration of tablets was wireless tethering with desktop applications, because “this ‘new’ way of interaction with a user interface that is so much more intuitive than having to nudge a mouse pointer across the screen.” Adobe released for developers the Adobe Photoshop Touch SKD which opens developers the door to think of clever ways to integrate a tethered device. For example, it could serve as a color palette where swiping blends colors together. The resulting colors then can be returned to the host application. EDIT: Adobe recently me an announcement via eMail to update Photoshop and get complement software from the AppStore. Pity I roll with Android.

The third possibility is using tablets as cloud-clients. nVidia has already a Flash-based technology for combining collaboration with Tegra-powered tablets called “Studiopass”: It lets you upload, stream, annotate and comment in real time on video files with others. It is also possible to use the tablet as intuitive virtual viewfinder that’s connected to a render farm that returns within a few seconds the rendered image to the device. And, surprise, Studiopass is built on Flash.

Just when Neil got glowing eyes and popped up a slide in purest nVidia’s corporat0- design reading “Super” Computing, the time was up. Close call!

Baking Light

François takes a pic,
originally uploaded by Phil Strahl.

I was too lazy to cross the street for Approaching a CG Production so I stayed in the König-Karl-Halle for Bakery Relight by Thomas Driemeyer, Emmanuel Turquin and François Belot from The Bakery who talked about their company and their software, Relight. I was a bit distracted by the amount of people swiping away on their iPhones, Joseph Olin among them, while François outlined the history of the company in southern France. I only got to hear the last few words, “Sand, Beaches and Girls!”, although what I understood as “Beaches” could have been some other word. Or maybe it just was the thick French accent and the tendency to pronounce some words French, or switch to French vocabularies entirely.

Relight is essentially a tool for lighting and shading shots and to develop looks, realistic as well as artistic. “This tool was designed by artists and not by engineers” they noted and indeed: As Emmanuel showed a quick overview of the interface it seemed quite straightforward, a bit like Katana. “You don’t need to make a break every time you want to render a change on your way to the final shot, you always work with the final and that really fast.” Even little comfort features are implemented, like solo-ing certain light sources.

It plugs directly into Maya, 3dsmax and XSI, it comes with its own hair and fur system that is created at render-time by its production renderer that produces really good and clean images. The renderer itself is a Reyes-like rendering algorithm, iterative and optimized for fast feedback in production quality. But this need for heavy-duty-caching takes its toll on memory consumption I guess.

Once the scene’s visible geometry is cached by renderer (many millions of polygons take a couple of minutes) Relight unfolds its performance. Lighting and rendering a forest-scene with hundreds of trees and atmospherics only took a minute on an average notebook computer — impressive. And once the geometry and the first lighting pass is cached, the renderer gets even faster with every change. It keeps track of what has been touched or adjusted and only updates the dependencies, what had been affected by the change; and nothing more. “Depending on how many processors you use, the suite scales pretty well accordingly.” Even motion-blur and DOF (depth of field) are fast.

Another big thing of Relight are point clouds for quickly carrying out calculations that would bring a raytracer too its knees, memory wise. Point clouds (or disk-clouds to be more precise) are independent from the underlying geometry and make it possible, for example, to render ambient occlusion in fur stunningly fast. But ambient occlusion is only one application. Point clouds can be used for glossy reflections, environment light (from HDRIs, for example), area-lights or sub-surface-scattering.

Towards the end, François added that since Relight provides feedback so fast, it is a great tool for budding lighting artists and there are many collaborations with schools. Then he closed with a sentence in thick French accent that sounded to me like “Thank you very much for your invention!”. You’re welcome!

Arnold, the Tracer

Whereas The Bakery‘s Relight tries as much as Pixar’s RenderMan to cheat around raytracing, Sony Pictures Imageworks has a completely opposite approach with their proprietary renderer Arnold. The lecture by Marcos Fajandro and Larry Gritz Path Tracing and Unbiased Rendering started off by comparing rasterizer/Reyes renderers and ray-tracing.

Of course, Reyes depends on shadow maps and offers no ray-tracing but can be insanely fast, whereas a pure ray-tracer craps out completely with too complex geometry (see the issues with Davy Jones as discussed on the FMX/08) and/or/like hair.

So what do you do? Integrate some limited raytracing into a Reyes renderer or hack rasterizing & (deep)shadow functionality into a raytracer? Both options aren’t exactly irresistible. So the folks at the Spanish company Solid Angle, I haven’t heard of ever before, developed Arnold.

The first feature film rendered with Arnold was the, in my opinion terrible, Monster House and now since a couple of years it has become Sony Pictures Imagework’s only renderer.

Arnold is based physically on the Monte-Carlo methods, it uses no storage for lighting and global illumination information, has only one quality knob to tweak on (namely the sampling) and outputs the final in a single pass. Yes, Arnold is a ray-tracer par excellence and traces millions of rays. “And when you trace so many rays you need to make the rays really fast”. Further, Arnold sports networked, programmable shaders, subdivision surfaces and yet can handle “hundreds of millions of triangles and hair splines” and virtually “hundreds of gigabytes of texture-maps”. Impressive!

Since everything (everything!) is ray-traced, the final image is optically seamless. There’s no need for say, a shadow pass or motion vectors. My worst fear of such a renderer is that it probably takes ages just to get a feedback of lighting changes, but that is not the case: An example video showed a car in Maya in the viewport whose camera was turned around it. As soon as the operator let it go, Arnold kicked it and its interactive mode rendered the selected region in big blocks, then smaller blocks and even smaller blocks until you could make out the raytraced details of car paint and reflections. This refinement process took not more that two seconds.

“In the end what matters is not how long you wait for your beauty-rendering but how long you wait for feedback of each iteration in the (re-)lighting of a shot. Since an artists costs you approximately $ 40 an hour, this way of saving time saves you a big amount of money and the artist downtime between iterations” Marcos explained. I heard several mental kaching-sounds in the audience.

Making raytraced motion-blur efficient is also important for the final rendering. A fully lit and shaded scene with production-quality deformation motion-blur rendered only 15% longer than without. Arnold even is proficient when it comes to volume renderings.

“The joy of Arnold is, that everything just clicks together seamlessly when you raytrace everything in your image. If you use different renderes for different tasks, even simple things like shadows can be a problem, like shadows in volumes. With Arnold there’s no catch — you get the whole package in one go with ambient occlusion, motion blur, soft shadows on motion blur, caustics, etc. ” Marcos summed up.

And just as joyful is instancing of geometry that gets loaded into the memory once for whatever number of instances. One shot of Alice in Wonderland had 61 million triangles and could be rendered in a single thread’s 15 hrs which boils down to an hour or less in a farm.

“Still there is lots of room for future improvements” Marcos concluded before Larry took over. Larry talked in more detail about Sony’s way of VFX and relighting using the upcoming Smurfs movie as an example.

For every shot with CG-integration, they shot a HDRI of the set with an incredibly expensive Spheron camera. In fact, not just one panorama, but two in different heights so with the set survey data it is possible to calculate and to recreate the scene geometry via triangulation roughly in CG. Then, for quality reasons, the light sources and instruments in the panorama get replaced by Arnold lights and erased from the HDRI panorama via Katana. The actual plate from the camera is projected onto the low-poly scene geometry, then the HDRI panoramas get also projected into the scene. “It’s kinda the same what was done for Benjamin Button” Larry added. Now the CG-characters are imported into the scene and get their lighting and bounces from the surfaces that the plate and panorama was projected onto.

As mentioned above, Arnold has an open shading language. “Traditionally, shaders are black boxes to the renderer that just return some values to it; their units are sloppy oftentimes and their underlying C/C++ code can also crash the renderer altogether. With an open shading language, we don’t have these issues.” Instead of returning color values to the renderer, the Arnold shaders compute so called “closures”, descriptions of how the surface will react to light, and pass these numbers on to the renderer, which can decide what to do with them. “This is also 20% faster!” Larry added happily.

What working with these “closures” exactly meant and what it had to do with ray-budgeting and Multi-Importance-Sampling (MIS) for the Monte-Carlo raytracing I did not fully grasp but Larry had some links (and his email address) on the subject, in case anyone was interested. I reproduce them here for the same purpose:

• http://bit.ly/s10shaders
• http://opensource.imageworks.com
• lg AT imageworks.com

Some features of Arnold like sub-surface-scattering are performed with point clouds on-the-fly, although the renderer still employs Monte-Carlo methods when point clouds won’t suffice. Since Arnold is proprietary, Imageworks developed a plug-in for XSI themselves and a very basic one for Maya.

But why did Sony Pictures Imageworks settle for just Arnold a few years back? “The pass management grew so complex and bloated, it was hacks upon hacks, that overwhelmed the mental capacity of the TDs. Now it’s back to just having a single pass. Lighting got so much faster and in the end. Lighting and rendering a single pass still is faster than rendering lots of different passes and trying to get them working together.” Larry explained, and Marcos added: “When you reduce time somewhere in the process, however, somewhere somebody uses the new freedom to add more complexity.”

Currently Arnold uses only the CPU despite the trend of making everything CUDA- and hence GPU-compatible. “Porting would be complex and time consuming and in the end wouldn’t speed up things significantly, so we spend the efforts to optimize the CPU code instead. Further, the terabytes of textures would need to be streamed to the GPUs as well…” Larry summed up the bottleneck situation.

“We still have a couple of minutes left so I’m gonna show you the Green Lantern trailer. Everything was rendered with Arnold.”

Fun fact: The plate at the end of the trailer with the title and release date also had a rather small line reading “Also playing in 2D theaters.” Like it or not: Stereo has finally arrived and it is here to stay. Deal with it!

And here’s a LaTeX-set presentation on Importance Sampling for Monte Carlo Ray Tracing from 2006 with lots of pretty pictures (and some equations).

Coffee!

Haus der Wirtschaft,
originally uploaded by Phil Strahl.

The few hours I was asleep took their toll and I was wasted for the break. I schlepped my bag and camera to a long queue in front of Starbucks where I got me a motivational Caramel Macchiato so I had the energy to look for food. I found an Asian noodles stand in a subway station and made my way back to the Haus der Wirtschaft, munching on fresh cooked vegetables. The remaining ten minutes I had an espresso in the showroom of the FMX. When I felt ready for some more lectures and headed into the König-Karl-Halle and was surprised to see all the good seats taken. So I settled for a suboptimal seat next to a German student who was constantly eating or drinking something and merrily ignoring the ban on recording devices. Funny thing though was that he thought I didn’t speak German and I was in no mood to shatter his belief. So I sat there with Ophelia, my notebook, in her bag on my lap and waited eagerly for the next lecture to take place.

Render de Janeiro

Building, Lighting & Rendering “Rio” by Blue Sky’s Andrew Beddini was up next. Andrew somehow reminded me of a friendly Ben Stiller character and his engaging lecture was a pleasure to listen to without once the need to close my eyes for just a couple of hours. Or maybe it was because I was so close sitting to the pumped-up loudspeakers that I feared my ears would pop as they rolled the FMX-trailer.

Blue Sky, founded in 1987, is a veteran of the industry, they even took part in creating CG sequences in Disney’s original TRON. The next milestone was the animated short Bunny in 1998 I remember seeing at the Ars Electronica that time. I even remember that it was the first animation to employ the nowadays obscurely named Radiosity.

People really like to talk about their renderers I realized, as Andrew started summarizing the features of their proprietary renderer with the featureless name CGI Studio, or just Studio. On a rendering from 1993 Andy demonstrated the features their raytracer could produce back then which they still use today. Some of its features used for Rio were secondary rays, true radiosity, raytraced soft shadows, tesselation of beziér patches, a procedural shading pipeline and the possibility to use procedures for much more than just shaders. Rio is a feature of 1800 shots, and 80% of it had to be done in just 5 months so the focus was on creating a pipeline that was fast and efficient.

Character lighting was one of the main and foremost concerns of the production. And “good lighting needs good development.” The character of Susan with fair skin, glasses and big eyes was the first testing ground for plausible shading and lighting. First they applied the sub-surface-scattering-shader to her skin and realized, oh boy, that it looked not really convincing. So the first thing was to implement a density adjustment into it, then light transmittance through the skin, radiosity and a subtle but necessary secondary transmittance model that made the skin softer still.

Eyes behind glasses were another challenge because “eyes are critical for emotion. You really want to get the eyes right first. If they are off, if the audience doesn’t buy it, then all the other efforts are in vain, so get the eyes right!”. First they raytraced the eyes and the glasses but the physically realistic look did not work with the stylized character design. For example, the shadowing of the glasses under the rim was just too dark with the realistic refractive index of 1.5, only 1.1 was just about right. Can you do this? Should you do this? Andrew made it heard: “Break reality to make it work for you!” What was behind the glasses was rendered separately with a plate of the background to yield realistic refractions of it, as for the reflections, the reverse angle shot was used ever so subtly. But still the eyes didn’t look alive, so Blue Sky went the whole nine yards and rendered for the eyes a UV pass, a reflection pass, an object pass so the highlight could be adjusted in Nuke accordingly. This finally gave the eyes that certain something that was missing and was tweakable in every shot.

The set creation of the favelas was a difficult task as well. After the first color studies the assets were created with low complexity in a modular fashion and could be combined to a very organic whole according to a plan of the set’s final layout. “If you want to sell a set to your director render it in gray with just the ambient occlusion and he’ll love it!” Andrew joked. “There were absolutely no texture maps involved, everything was done procedurally with using the world space coordinates. Also we don’t need a level-of-detail (LOD) system.”

With just one building block, the lighting department instanced the geometry for lighting and mood tests of what the location might look in broad daylight, in the afternoon, at night and with atmospheric effects.

Raytracing and motion-blur — another dreaded combination of mine. But to tackle that problem Blue Sky employed a lot of tricks that made their lives easier. One of those was modulating the render-resolution accordingly via script: “At some point you can’t tell whether a 2k image was blurred or a 1k image was blurred, so if the renderer dials down the resolution you effectively are four times faster.” Another trick was using Nuke’s excellent vector blur: The camera and a Z-Depth channel from Maya got imported into Nuke and was blurred there in post.

A slide depicting the favela at night appeard on the screen. “In this particular scene we had about ten thousand light sources” Andy spoke and paused for emphasis. Ten thousand light sources with raytraced shadows?! “The thing is,” he continued “we use the same pool for all shadows, they get calculated at the same time. So if we’re having one light source or ten thousand only adds 10 to 15 % to the overall render time.”

Since the protagonists in Rio are birds, the sky is an important part of the film as well. “About 30% of the picture is sky!”. So how do you get a beautiful art-directed sky? The traditional answer is to have talented matte-painters. “The problem is, that we had four to five matte painters but hundreds of skies to paint. So this wasn’t feasible. Further the feature is in stereo, so…” he switched to the next slide depicting various clouds on a black background “…we had our art director paint a style guide for clouds we then built in 3D with volumes. We rendered each cloud with surface normals from 38 different camera tilts so you had almost any angle. Then we imported those onto planes in Nuke and populated the sky in the 3d-space there with the clouds.” The normals made it possible to vertex-light the cloud planes in Nuke. “This is really fast and you can churn out skies quickly. We did this for an entire sequence at once, not only for a single shot.”
But sometimes you need hero clouds, especially when they needed to have a volume and depth or some advanced lighting effects, like transmission on the edges of a key light. These clouds got individually modeled and rendered. The atmosphere-gradient with the sun was also done as a dome in Nuke and when you throw everything together — voilá — there’s your final sky! This sky then got rendered in Nuke and was used on a plane in Maya for the reflections on the water.

Generating a vista of Rio de Janeiro, especially a stylized version that still looks as photoreal as possible was the next challenge on Rio. The foundation for the environment was survey data of the topography that got artistically adjusted with some liberties. Still, it was built to scale so the renderer would deliver correct results of atmospheric effects, for example.

The vegetation was, like almost everything in Rio, procedural with implicit surfaces. But here the procedural approach posed some difficulties: it is hard getting a procedure to a single point. As for the shading, two world space procedural textures were created, one for the granite and one for the plants. A simple rule that depends on the steepness of the topology then blended between the two, so vertical walls would have no vegetation and soft slopes would be fully covered in trees. Finally, the city’s buildings were roughly modeled, but also were designed for procedurally driven variety: They could have differently spaced windows, doors, floor-segmentation, surfaces and rooftop structures.

What surprised me the most was the way the lighting was defined and carried out: It was script-driven and those scripts were simple text-files the production edited with NEdit in Perl and Python. “This is great because the files are small and portable, you can send them securely via email, and you don’t need to open Maya just to twist a light a bit to one side.” Blue Sky even has an interactive front-end called “Quick Render” to render models and light situations without waiting for Maya to finally launch. And there was more: “Light sources are not defined in the world-space but in camera-space,” instead of XYZ they are described by values for rotation, elevation and distance from the camera. Again, you don’t need Maya to adjust a light; creating a consistent rim-light can be done with a few lines in a text-editor easily.

Finally, even the atmospherics were rendered in favor of a Z-pass. “A Z-pass always has artifacts along the edges so you need to render it really big.” And once you have time for that and to add fog & haze in post, you can render physically much more plausible (and within the scene reflected) atmospherics.

A Hairy Subject

We only had a few minutes to let sink in what was heard before Mohit Kallianpur from Disney Animation Studios continued with his lecture of Untangling “Tangled” where he was the Look & Lighting supervisor and described the history of the look of the movie.

In 2007 the movie had a much darker, browner tone to its concept art until John Lasseter intervened and pointed the production in a more colorful and saturated direction. Still, that was relatively late in the pre-production stage and there was not that much time left to get the movie done. So Mohit got down to the root of art direction and formulated three principles: Stylized shapes, illustrative colors and believable textures.

Shapes

Then the research begun by watching and analyzing the shapes, colors and appeal of the old Disney classics Snowhite, Pinocchio and Cinderella. Especially the latter had a certain shape language of flowing curves visible in almost every shot, a graceful harmony. Moreover, a set of signature shapes of that movie was collected consisting of various bell-shapes, s-curves and leafs. Everything in the film would follow these shapes, even the canopy of the trees.

Based on a very impressionist and rough mood-painting that followed the shape-guide, the team produced a full CG version of it as reference. This made it obvious that the language of shapes worked well, but the painterly appeal of the surfaces was too stylized. So the world needed to have believable textures, not rely on impressionist suggestions of detail.

Architecture

The architecture of Tangled was influenced by European cities, Disneyland (!) and Pinocchio: Everything should be small, friendly and approachable. The buildings are not tall and they flare out in a curve, they appear even chunky and beefy with no sharp corners and a very organic and hand-built feel. And like this Mohit wanted the architecture on Tangled to look: old and used but not decrepit or dirty.

Color

But what does “illustrative color” now mean exactly? The production settled for a lush saturated palette and always a play of warm against cool: If the light is cool, then the shadows should be warm and vice versa.

Legally Blonde

Yet the most intricate and most critical task was to get the look of Rapunzel’s hair perfect. Much like the sky in Rio, the hair in Tangled was a character by itself. The first tests showed that the hair shaders the studio had developed so far didn’t do justice to blonde hair. At all. So research needed to be done and a hair model was found and her hair photographed in probably any appearance (even wet) and every possible lighting situation. Also commercials of hair care products served as a valuable reference “since they propagate the ideal on their packaging we wanted to recreate in the film.” Even a PhD student did extensive research on the subject and in the end a feasible shader was programmed:

First of all, hair comes in individual strands and strands that make up a volume of hair. Moreover, these strands have a top, a bottom and a body with an uneven surface, none of which is like the cylinder-representation of hair we all used to work with. So hair has a specular reflection, so far so good. And for black hair that’s usually all you see. Light hair also has a sub-specular portion that is a broader highlight in the color of the hair, whereas the specular highlight has the color of the light source. Then hair has a transmission value when lit from behind, and multiple scattering is important for light colored hair. Still that’s not enough because a hair rarely comes singly: There’s also a volume diffuse portion, that is backward scattering of the light and volume transmission, which scatters away from the light source through the volume.

Now if you add up those five components it looks good but not perfect, a certain richness is missing. This “richness” comes from the light bounced back onto the hair and ambient occlusion. Then, and only then, you are rewarded with beautiful blonde hair. But for eyebrows, fur and eyelashes you still need a different shader because the blonde shader needs a volume to work with after all.

Appeal

And another question: What constitutes appeal? What makes a person look appealing in a movie? How to light a character to look appealing? Mohit dug deep once again and looked through lots and lots of glamor-shots of actresses and actors of Hollywood’s golden age in search for unifying principles. And his answer is simple: “Cheat!”

The starlets on photos (and in movies) were always lit by a soft light, no matter what the rest of their environment looked like. Then there needed to be hue in the shadows, the dreaded “graying” of multiplying occlusion with diffuse passes makes characters look sickly. Since the eyes are the window to the soul, they needed special attention and always a specular reflection, no matter what. And as painters suggested, the cavities of mouth and nostrils should not go completely black but into a warm darkness. And unappealing colors (usually green bounces of leaves and grass) also make a character look weak and sick, so they had to cheat there as well. Finally a subtle bloom on the highlights never hurt anyone.

Look

On Tangled there were 19 look development artists that in the end produced over a thousand paintings of looks. And lighting was important to elevate the mood that was already there by narration, staging and framing. As Mohit showed some examples of lighting I could not help but remember the words of my mentor, Fraser Maclean as Mohit uttered them as well: “Put light where you need it, not where it realistically would be.” Indeed, the progression of shots he showed were lacking any continuity of the origin of the lights. “And here… well I don’t know where the light is supposed to be coming from, there is no window there” …but still it works perfectly in each shot. It went so far, that even the murals in the tower could be toned down or changed in opacity on a shot-by-shot basis. “Light shapes add to the drama.”

And so does saturation. In dramatic sequences the saturation got sucked out of the pictures, only to return almost fully later on.

Art-Directed Trees

Once you can art-direct everything, you have to art-direct everything. The R&D department provided a simple tool for Maya that allowed the modeling artist to draw a couple of curves and the plug-in would make a tree with branches out of it. Moreover, they also had a tool that would grow leaves into a pre-defined canopy-shape which was really fun to watch.
And since there are so many trees in a forest, the models were also switched to “brickmaps”, a RenderMan-term for low-res voxel-representations of a model, if far enough away and automated “stochastic pruning”, which meant that depending on the distance to the camera, not visible leaves would be automatically switched with low-poly models or removed entirely.

“In the end we generated too many trees and had to hide them in haze or atmosphere in post” Mohit admitted. That really was new to me, having accidentally too many in the final picture of something so complex such as trees!

Tangled was rendered with RenderMan (finally another Reyes renderer today, I was almost worried!), had 1380 shots and 55 lighting artists which, in sum, resulted in 9.01 million hours of a single lighting thread. “That’s more than 1028 years”. Ah, statistics. It’s like I tell you now that this blog post is already some 6000-odd words long.

Physically Based Shading

Ben Snow,
originally uploaded by Phil Strahl.

I used the short break to get back close to the front in row 2 and to empty my can of Starbucks “Doubleshot Espresso” to be all up an ready for Ben Snow’s and Christophe Héry’s Physically Based Shading at ILM¹.

Ben showed some definitions at first outlining GI, IBL and HDRI. “We’re not trying to reproduce reality,” Ben made clear, “but filmed reality” and described a little ILM’s history in their attempts to achieve this goal over the years.

The practice of trying to capture as much information from the set grew over the years in scope and professionalism as well. In the 1990′s they started filming and photographing light probes on set, 18% gray balls, to recreate the light later in the computer. But soon they realized that was not enough. Along came six photos in each direction for cube-maps and then the familiar chrome ball, a technique I never quite got to work for my own projects. Indeed, the chrome sphere’s reflections were rather low-res and you needed to paint out the photographer every time as well.
So in the “early days” ILM made heavy use of texture maps with painted in highlights and shadows, for shading they employed the Cook-Torrance specular model. Their light rigs were also pretty basic but could handle real world situations already rather well. Occasionally shadows grew really dark and some ways to cheat were e.g. spot lights or churning down the overall shadow opacity².

Then came Perl Harbor when they really needed to crank their depicted reality up a notch since ambient occlusion wasn’t enough. So they developed what they called Ambient Environment Lighting, which essentially is creating a pass for the ambient lighting via ambient occlusion: Rays are cast in a hemisphere around the surface normals, then a number of rays hitting other surfaces dictates the occlusion; and a pre-pass is done to calculate the average direction of the light. At least that’s what I copied from Ben’s slides. The film was a milestone nevertheless because Michael Bay, the director, was not able to tell the difference between CG and live-action-footage anymore.

What happened after Perl Harbor was boosting the quality consistently further. 8-bit images had served their purpose well (including in Perl Harbor!), but the need and time asked for floating point precision. Also, no mirrored balls would be photographed anymore in favor of photographed panoramas.

So a couple of years afterward came Iron Man and asked for the realistic depiction of, well, iron and metal that needed to match the practical suits on set. And a thing that had been troubling the folks at ILM was anisotropic highlights that appear on brushed metal.

Further, a new approach for taking HDRI panoramas was employed and meant taking a series of photographs from a tripod in all directions, although (as it appeared to me) not in a truly high dynamic range but only covering two two exposure brackets — too little from my experience but still it worked for them.

New Frontiers

And just as they thought they had mastered metal, along came Terminator: Salvation where it was impossible to cheat any longer and they had to move to a new paradigm for lighting and shading in RenderMan. ILM’s goal was to get a simpler, more intuitive and physically based system of lighting and rendering:

The quality of shading and lighting requred a BRDF-model that not only looked right but also acted physically correct in terms of energy conservation. In short this means, that the rougher a material is, the weaker is highlight gets, otherwise the energy (= light) reflected would be bigger than the light received — impossible³.

That also meant a normalized specular highlights with a more physically plausible specular falloff, so the further away a light source is, the weaker the specular highlight gets, depending on the roughness or the “Normalized Importance Falloff”: The intensity of the highlight falls off on rougher surfaces. For tight specular highlights like chrome or mirrors the light source has to get a long way away before dimming. The broader the speculars are, the more quickly they will dim. For this to work the light needs to have a physical size in the system, so no more point-lights or directional lights at ILM

“This was hard for everyone to adjust to and Christophe and I remember some very passionately fought holy wars” Ben remembered.

On the set of Terminator: Salvation the chrome spheres were back, but differently. Now they were moved and shot in motion so their reflection could be used on moving models. “We still don’t have a great way of capturing HDR moving images,” Ben continued in front of a turntable of a T-800 in front of a steelworks plate, “instead we shoot HDRIs with stable lighting and we also shoot our chrome spheres so we still get our FX, strobes, sparks, etc. And on top of that we applied some pyro elements shot on film and used them as reflections or area-lights in the scene.”

Iron Man 2

“We now had our tools that worked, they weren’t really mature but technically they were robust” Ben noted. For image based lighting ILM then used a graphical tool, the Environments Browser, to quickly define light sources within the image so they could be recreated with area lights. The match-move of a shot creates the environment and the HDRI panoramas get projected onto the geometry and you basically end up with a HDRI-mapped recreation of the set environment in 3d. That enabled the artists to render dynamic HDRIs from any position in the set to be used in image based lighting of the digital characters.

IBL via HDRI

But Image Based Lighting also poses some risks or at least things to watch out for. An important thing to be aware of is that IBL-lights are treated as infinitely away point-lights from the scene. A fact, that comes distractingly into play when CG-lights with a discrete position within the scene are meant to work alongside an IBL dome. Then it really takes an experienced photographer and visual effects team to produce good and usable HDRIs from the set which is essential for this approach. Last but not least the pipeline should support the the floating-point nature of HDRIs or else one might run the risk of losing dynamic range along the process when editing HDRIs.

On set ILM still photographs chrome and grey spheres but only as references. This only works properly if the film crew is in the habit as well and does not treat those shots lightly or worse, forgets about them. Since the VFX team later needs to match the lighting of the shot everybody liked on set, those references should be shot immediately after the director yelled “cut!”. “You wanna make sure that your spheres are as big as possible in the frame and you wanna make sure they are in the right spot. And you don’t wanna shadow or be reflected in the sphere” Ben reminded. Sometimes when the spheres are moved like an object though the scene, it is advisable to take some static sphere footage as well.

“On the set as soon as they say We’ve got it! then off you go and get the references.” These references, of course, should only be shot for shots that will have CG portions but if there is ever any doubt that a certain shot definitely will not require CG, one should still capture the references to be on the safe side.

“And be serious about it. If you don’t take this seriously nobody else on the crew will” Ben shared his experience with the audience and switched to his last presentation slide, titled “How we capture HDRIs” which I shall reproduce here:

• Canon 1Ds Mk3 with Sigma 8mm fisheye lens
• Nodal Ninja & Tripod
• Remote shutter trigger
• 0.6 ND (2-stop) filter (depending on how bright it is)
• 7 exposures, 3 stops apart
• Direct sun f/16, ISO 100, center exposure 1/32 sec

After a quick slide of acknowledgements Ben handed over to Christophe who would once again talk in a little more detail about the math-laden background behind the presented concepts.

Since my senior-high maths-teacher sucked out all the fun I ever had with mathematics, I wasn’t in the mood of trying to follow Christophe’s every word but I got the general idea:

Calculating reflections and IBL with the Monte Carlo approach requires a high amount of samples for a clean picture. Since the distribution of the rays is random, you end up calculating a lot of stuff you will not really see in your final rendering. So he came up with MIS, Multi Importance Sampling, that takes into account what the camera sees, that the light illuminates and identifies an area in the lighting-dome where rays have a probability to affect the final rendering, such bright portions in the HDRI used for IBL. With the same number of samples used for rendering, which are only weighted differently, in the end compose a much more pleasing result because you only calculate what you need.

To implement this into the renderer, the BRDFs and lights need to provide eval() and sample() methods, where eval() returns a color and a pdf for a given input direction and sample() that returns an array of directions and pdfs: For instance, in a dome IBL situation, these will be the vectors to the bright spots in the image.
I just copied this from a slide of Christophe’s presentation, so don’t ask me what it all means. The only thing I am pretty sure is that pdf in this context stands for probability distribution function and not for Adobe’s favorite way of storing their user guides.

ILM’s BRDF specification has to follow a number of principles too: The shaders now must be normalized e.g. energy conserving, they must “substitute” to ILM’s trusted but old Cook-Torrance model, they should be anisotropy aware and, as always, efficiently computed.

Their solution now is D-BRDF based on an yet unpublished paper by Michael Ashikhmin and Simon Premoze; the Beckmann distribution⁴ and no masking: the reciprocity term is simply 4.0 * V.H * max(L.N, V.N).

For the nerds among you I even noted the links Christophe’s presentation closed with:

• Eric Veach’s paper⁵
• Last year’s SIGGRAPH course
• D-BRDF

You’re welcome. Now where is my mind?

After this heavy information laden day I skipped nVidia’s panel discussion and went straight home to type up this blog post. who would have thought that it would take me effectively over two weeks to finish it?

Post is Prep

My Access Pass,
originally uploaded by Phil Strahl.

Still a bit drowsy I planted my ass in the front row of the König-Karl Halle and knew I wouldn’t be getting up for a long time, not even for Pixar’s Career Gears (they don’t need compositors, I got the message last years). So at 11 a.m. “The Role of Visualization in the VFX Production” by Kevin Todd Haug, VFX Designer at FX Cartel and Ron Frankel, president of Proof Inc. begun.

And it began with some heavy, uncommented statements right in the beginning that made me gulp a little and sit there a bit shocked and concerned.

• Post is Prep.
• Compositing is dead.
• Avatar cheated.

The speakers started off with describing the status quo, that currently everything except the production itself happens in a purely digital environment, although the direction towards a fully digital approach is present also in the day-to-day life of the film-making process on set. Yet this reality asks for a paradigm shift towards the non-linear workflow of pre-production on set as well.

“All the steps from pre-production towards the reality of the live-action set are a question of creative communication.”

There have always been paradigm shifts within the VFX industry, such as the switch from analogue to digital in post production in the past twenty years. Now we are headed towards having the virtual reality (VR) of the feature not only available in pre-production and post production but also in the production itself.

Some prominent features already have arrived at that step. Whereas James Cameron used these previsualization techniques to see the virtual set through the camera while shooting, Tim Burton was offered a similar approach on the stage of Alice, yet “he just turned the screen away and preferred looking at the green-screen footage.”

“Doing an animation paradigm doesn’t solve the problem, doesn’t treat the actors in a respectful way. The question is how to bring everything into a virtual world. And what does that mean for us VFX people? [...] Most things belong in post, you shouldn’t do matte paintings afterward, they need to be known by the director before the principal [shoot], so it needs to happen in prep.” In fact quite a lot of assets get worked on (and finished) in preproduction, not only matte-paintings, but also models, textures, characters and so on. So why not use everything that’s already there on set?

Tod and Ron then showed a hands-on example of Conan or as they jokingly put it: “The adventures in Low-Budget-Land”. It was a shoot of a sci-fi-esque sequence set in the Prohodna Cave, an enormous cave near Karlukovo village in Bulgaria. “This cave is hundreds of miles away from your local anything“, so shooting directly on location would’ve been way too expensive. So instead of wasting loads of money bringing the set to the cave they brought the cave to the set. The cave was recreated in VR and could be used as a virtual set.

To make this work, position and settings of the live action camera needed to be tracked in real time. Looking through the camera means looking at your actors and a replacement of the green screen with the virtual set. This allows the DP to set light, angles & composition of the live action parts much better because s/he always has a near-final background when looking through the lens without the need of eventual re-lighting in post.

“It’s there to enable a creative dialog instead of people just guessing what to do and how to do it. It’s all about the discussion, not about the technical tools itself. So DP and CG artists can work together in both prep and on-set and eventually all boundaries collapse into one united effort from the beginning.”

Then the statements of before flashed across the big screen again, but this time with remarks that evaporated my concerns.

Post is Prep

Entrance Hall,
originally uploaded by Phil Strahl.

The amount of work in post has no business in being in this environment. The whole point of being a Big Company assumes that you will bring a mass of problems from set to make them solve them — for a price. And that will change. The decisions will be made on set, hopefully diminishing a lot of problems. So post production will be much about “making it look nice” instead of “doing somebody else’s dirty laundry” as I like to express it.

Compositing is dead

Boy, that was a downer to me the first time I read it. Not so much a downer as “No, you won’t get a pony!”, it was much more like “There is no spoon.” Luckily Tod and Ron cleared it up what they inferred by that bold expression.

“Fixing a problem by ‘painting something out’ is not an option anymore. Since there’s an unstoppable shift towards stereoscopic features happening you have to make everything to work in stereo. And that makes a big difference when setting up a pipeline. Instead of everybody being a tiny cogwheel in the machinery, you will have more high-level artists working on shots from start to finish. And those shots will be cool!”

Avatar cheated

“Avatar isn’t really film-making. Rather it’s an animated movie with people who aren’t animators in funny suits. Essentially it is an animated movie with some live action in between.”

Wow.

Somebody really said it. But since there was all this pro-Avatar bias in the media and the industry I got a first glimpse of how the tremendous efforts of the making of Avatar where kind of equalized by the statements of some speakers.

“Everything on [Avatar ] was really expensive and if you do it again you get another Toy Story. And Pixar is way more efficient than Cameron. That paradigm just won’t work again.” And it is true: Everything on the set of Avatar only works in that particular stage and location. But in the day-to-day life of making movies (especially cost-effective) you need your equipment robust enough to travel. “The longer you stay in one location, the more you fuck it up [because] film crews really have a tendency to use up locations.”

I used the term “virtual set” quite a lot so far but Ron stressed that there is no such thing as a virtual set. “Since it is built somewhere it is real. In the end everything that’s being recorded is virtual. The bottom line is what’s on screen in the end and that’s what everybody’s there for.”

Personally I didn’t see anything strikingly new to this, although the quality and speed of displaying the virtual set though the video-playout improved over the years. But what was really impressive to me was employing (rather) simple camera projections onto simple geometry to make it possible to wander through concept art and matte paintings literally in one’s office.

And I even learned that there is a Society of Previsualization out there. Who knew?

The New Art of Virtual MoneyMoviemaking

Without much time in between Maurice Patel from Autodesk continued the previs-morning of that day. Maurice started of stressing that it is the tools you need to enable interactivity such as real-time motion capture, real-time processing and real-time playback. Since teams need to work closer together and need to collaborate bringing everything together is of key importance, since “communication is the most powerful in visualization”.

In today’s digital production there is a wide range of different elements that need to work together like art, mo-cap, previs, CGI, rendering and what you get from the practical production on location. Not until post you start putting everything together. And we all know by now that making changes in stereo is difficult if possible at all.

There’s a lot I didn’t note down what followed because in the end it was just an ad-laden lecture of Autodesk technologies and previs-services, probably nothing one wouldn’t find out by browsing their website.

What I understood the bottom-line was “Build the technology. Or buy it from Autodesk. Then work with different departments to implement it.”

Still: I see that there is a trend evolving in the industry to get things right from the beginning by employing various previs-techniques just to minimize the workload of “fixing it in post”.

Abyss to Avatar

The Slide,
originally uploaded by Phil Strahl.

Director and VFX legend John Bruno was talking about his role in the industry and being more or less the right hand of James Cameron in a number of films such as The Abyss, True Lies, Terminator 2: Judgment Day and last but definitely not least Avatar. In the talk he showed clips from each of the movies and explained some of the techniques they used. The face made of water from The Abyss for example was composited optically, in the original render it resembled a chrome-tentacle instead of being seemingly (and rather convincingly) made out of water.

Unfortunately this was another lecture where I wasn’t allowed to take some pictures so you have to endue the rather boring title slide I photographed. My apologies.

Disney’s Tools

The next lecture by Disney’s Andy Hendrickson was titled a bit clunk “Blend (Art+Science) = Technology at Disney” but had some interesting aspects.

Andy presented in the beginning a typical Disney concept art in a purplely-brown and uplifting tone, depicting a stone tower and trees with a brook in the foreground, a waterfall and very picturesque rock-formations behind it for the upcoming feature Tangled, formerly known as Rapunzel. You can see the original artwork by clicking here.

The following hour Andy broke down the painting into several details such as the waterfall or the lighting which were created part by part to match the concept art as closely as possible. “At Disney it’s not so much about rendering something photo-real, it’s more about staying true to the artwork. And for that we need different technologies.”

Now check the concept art from above with the final CG-version which can be seen right in the beginning of this teaser right here on YouTube. Pretty darn close, huh?

“Our concept artist boasted that he could paint these artworks in the evening after having a six-pack in one or two hours, which would take us weeks to figure out on how to recreate them digitally.” Andy added with a smile.

Disney uses a combination of xGen (whatever this is), RenderMan and IRender. They also plan to release their old classics in stereo and have developed a technology that more or less automatically produces usable depth maps, however they also employ 3d-models to project the 2d imagery on.

Personally I am not so much a fan of Disney’s policies, economic decisions and certain aspects in their style, yet they decided to release some of their technology with open source, so that’s something new to the whole proprietary-focused industry.

So I recommend taking a look at Disney’s Open Source site for the Ptex texture mapping system that doesn’t need UVs (That’s a bingo!), and I strongly recommend browsing through Disney’s publications on that site as well.

You are not welcome here

Being a big fan of Neill Blomkamp’s District 9 and knowing that the movie’s VFX were comparably low-budget made me stay for the next presentation by Dan Kaufman’s “Inside District 9″.

So there was not much money for the VFX, yet their quality and integration into the wild live-action plates is stunning and seamless.

The design process went through many iterations, at first the shape and physiology of the aliens would allow a guy in a suit to double for an alien but this turned out to take out a lot of the anticipated realism because the audience would always be able to tell that, well, it was a guy in a suit, even if his face would have been fully CG. So the early maquettes¹ defined the overall style. The final appearance of the aliens featured a more rigid exoskeleton even on the face, yet these overlapping scales were attached to the underlying geometry and animating facial expressions was not that big of a challenge.

A lot of tests of various animals were made for the appearance of the eyes, in the end the team settled on a human pupil and iris but on a black eyeball.

The main part of the characters was performed with Maya. In order to minimize cloth simulations their garments were applied very tight-fitting on the characters so that only certain bits needed to be simulated as dangling on the hero characters. Further they had stickers and make-up applied as an additional pass which all helped in diversifying the aliens while keeping them least troublesome in post.

The compositing was done in Nuke and required a lot roto’n'paint as well as 3d-tracking and projections: Most of the shots in the movie were hand-held and had actors as stand-ins for the aliens to help the other actors as well as the cinematographer to know what was going on and where. Later, the stand-ins needed to be removed from the wild plates by geometry projections, sometimes requiring exhausting and complicated paint work.

And there’s one slide of the presentation I recorded, in my opinion the most important one:

Staying On Schedule And On Budget

Use the simplest approach that will achieve the goal. Plan the entire pipeline

• Build in flexibility
• Make it as foolproof as possible
• Assume things will go wrong and have alternate strategies
• Keep communication flowing

Work closely with director/production

• Discuss trade-offs
• Come up with alternatives that still achieve artistic/story intent

Now that’s what I am talking about!

Rolling Shutter?

Ben Grossmann,
originally uploaded by Phil Strahl.

Ben Grossmann from CafeFX jumped right into talking about the VFX on Martin Scorsese’s “Shutter Island”. I really liked the fact that many applied techniques were more traditional thanks to his way of movie making, nevertheless he was very open to new approaches in it. In the end there was a lot of forced perspective and model-work in the feature, composited either even in-camera or in Nuke.

There’s an excellent interview with Matthew Gratzner and Ben on fxguide that I recommend to you if you are interested in the whole making-of stuff behind Shutter Island, because the lecture itself was nothing more than a narrated feature, although there is one technique described that wasn’t in the interview:

Details and “tiles” of the sea crashing against New-England-shore-rocks were filmed from a crane on a location that “looked just right”. This location was shot from all angles, starting from looking directly down onto the shore until up to sky, producing what Ben called “salad bowls” of tiles that could be re-timed to match each other and be projected onto a semi-sphere in Nuke and be used in/for matte paintings.

When asked about the work with Martin Scorsese Ben told that it was incredibly exhausting but totally worth it, because everybody knew that he was a legendary director and knew exactly what he was doing. In the end, so Ben said, “Marty got 200% of the VFX he asked for. For free.” Incredible, how being a legend helps in keeping the costs down in some way.

Crosseyed Avatar

Before getting his grip on Nuke Christian Kaestner was first outlining the situation of the current trend with stereoscopic pictures: They are box-office hits (e.g. Avatar) and, what’s probably most important to the industry, limits piracy because you can’t download a feature in stereo and enjoy it at home as you would in the theater (yet!). So, like it or not, it’s here to stay.

But stereo is expensive. Not so much if you are dealing only with a live-action film totally shot in stereo, or a full CGI film in stereo, no, the spending gets out of proportions when you mix live-action and CGI. For example, match moving takes three and a half times as long as in a traditional show, compositing (especially scene salvage) gets at least twice as long and, alas, annoying.

The on-set experience of Avatar was extraordinary for it was being in production for more than four years. The set itself was more like a full studio with all its custom-developed technology for and by James Cameron. But he needed the time to plan for single shot in detail, and also in stereo depth because you can’t afford mistakes at that scale. The previsualizations he approved became “the Bible” for how shots needed to be carried out, not even minor differences between previz and final in, say, distant mountain in the matte-painting were allowed (“and Jim got quite a temper!”). So quality control passes needed to be reviewed and green-lighted as well which essentially were the keyed live-action shots with un-shaded geometry but a 100% correct and tight track for each eye.
Most of the shots were composited in Nuke and rendered with Pixar’s RenderMan. Because so many studios were working with the same assets, which also partly existed as animatronics on set, all of them needed to be matched in appearance. The most complex shot in Avatar was hg016_0065, Jake Sully rolling out of the carrier after arriving on Pandora with a bypassing mech: 250,000 files needed to be rendered and were comped in a Nuke script with over 3,500 nodes. “Yet it rendered on our farm in about ten minutes. Nuke is incredibly performant!”

Let’s get physical!

It is often (if not always) the case that even matched lenses and cameras differ in their color fidelity and lens warping. So what needs to be done in the first place is to get rid of the lens distortion by a lens correction node. Then the problems in color can be addressed which is the tricky part: Some areas in the image match, some don’t and in the making of Avatar “a group of skilled people was locked into a room without windows for many weeks to match those kind of shots by hand”.

At this point in the presentation Ben from The Foundry took over and showed some nodes of Ocula 2.1 to tackle exactly this kind of problems. Despite some of them not even being in beta, everything worked out just fine for the presentation.

First Ben showed how to create a disparity map by connecting a DisparityGenerate to a O_Solver node; nothing new here: The result still is an image whose pixel values in the first two channels describe how the pixels from image A need to be transformed to become image B. Ben then set the analyzer to a single frame and added a keyframe. One can now generate image B from image A and check it against the actual image B for errors and mis-alignments; although moving objects (such as actors) should be left out.
So if you can build a better image out of it, it works. The node even offers two options, “Normal” intended for motion and “Severe” which checks the edges, and I think it was in the O_newView or TuneDisparity node (provided the latter one even exists). The image generated from the disparity can be written to an EXR and then color matched with the “real” image from the other eye.

Unfortunately some flickering in can be observed, but there is a way to multisample the ColorMatcher by frame-blending the disparity for a couple of frames and/or by applying five or more ColorMatchers, each with slightly different block-sizes. Then merging them with each other by Plus, multiplying the result together and removing the original image. The result may appear a bit blocky, but it can be blurred and combined with the footage. The blurriness is almost invisible, yet this approach only works for footage and areas with low self-occlusions of the object. The Ocula 2.1 node, however, has now a multisampling-option for exactly that kind of merging a number of ColorMatchers together.

A similar approach can be made to perform depth grading, generating new virtual cameras that lie between the recorded eye-positions. Lastly there is also a DisparityViewer which shows the disparity vectors. Ideally they have the same length, are horizontal and have an offset by 180° in their angle.

Haus der Wirtschaft,
originally uploaded by Phil Strahl.

I was eager to first see The Foundry‘s presentations on Mari, their programming approach and a tech demonstration of the recently acquired Katana, I was so excited about last year in Sony’s presentation, but had no exact clue what it really was.

Paint that dinosaur!

Once arrived I got me a seat pretty close up front and was ready for their presentations to begin. Jack Greasley, who worked at Weta Digital on King Kong and Avatar and Zoe Lord, Senior Texture Artist on Avatar, presented Foundry’s upcoming texturing tool Mari.

Zoe, Jack, Bruno & Andy,
originally uploaded by Phil Strahl.

Weta needed a decent and perfomant tool because handling the characters of King Kong for Peter Jackson’s movie was a challenge itself. The monkey was huge and complicated and worked more like a complete set than just a character. And after learning about the tedious process, everyone at Weta knew, that Avatar would become just as exhausting and complicated like Kong — only multiplied hundreds of times.

The first version of Mari was took 16 months to develop and once it was running, it was constantly in production and used by the artists in production until up the current version. Although Mari is now internally in version five, for its public release Mari will launch as v1.0

Essentially Mari is intended as middleware between sculpting and animation, and effectively completely replacing Photoshop for a texture artist. Since this tool has been developed at Weta Digital for a couple of years now, the workflow in Mari is rather straight forward: You import hundreds of still images that can be manipulated inside the software and applied directly to the model you’re working on. You can perform 2D operations to your references such as cropping, color correcting but also warping or pinning. The performance is outstanding, the software handles well over a 100 2k-maps in realtime on a multi-million poly model, and here’s the best part, Mari can read and play back .obj-sequences of an animation so an artist can correct ugly stretching errors in the texture on the fly. this performance allows the artist to load and work on whole sets with moving objects and to create seamless textures across objects more easily. Did I mention that you can, of course, animate your textures when needed?
Mari is also capable of rendering occlusion-passes which can not just be multiplied to the color-maps with blending modes like in Photoshop, you can also use them as masks to paint dirtmaps.

“We realized that with other tools the artists were much more data wranglers than working creatively, setting up map-channels, importing files and shading networks and so on.” Mari offers the artist various tools and guides to work effectively and to avoid mistakes such as a protection system for edges or the possibility to use channels as masks.

I wanted to know about the exporting capabilities of the tool, like whether it was possible to export your shading network from Mari straight into Maya’s hypershade. “Unfortunately not, you will need to export your maps as separate TIFs, but the SDK is very open so you can have scripts in your pipeline that do that for you.”

Also displacement maps are currently only previewed in Mari as bump, yet The Foundry works closely with nVidia and ATI to add certain features.

And you’ll need at least a gig of free video memory to allow Mari to unfurl its glory.

6.1

Simon Robinson,
originally uploaded by Phil Strahl.

Simon Robinson continued with what’s new and cool with Nuke 6.1, such as the refined camera tracker or “stuff that’s not really rocket-science but still needed to be done like moving a camera while looking though it”; “GeoSelect” to select points of a point cloud; the modeler node that less you create geometry between points of a point cloud, which works a bit like in Boujou, but with the advantage of refining the corner over time yourself and have Nuke fix the track cleverly. This seems to be the way to go in terms of scene salvage in steroscopic productions, where you project your paintwork on such geometry in your scene. The WriteGeo node now also lets you export .fbx files, not only load them.

Nuke 6.1 features,
originally uploaded by Phil Strahl.

Another thing totally interesting is baseline camera correction, which I didn’t really grasp in the short presentation, but it appeared to let you stitch panoramas together and create 3d-vistas out of them, kinda like photogrammetry.

And in addition to Keylite and Primatte, Nuke will also support the Ultimatte keyer, so Nuke will have all three industry standard keyers available.

Simon also provided an outlook of further Nuke releases, mainly improvements in stereo roto & paint. When asked in what Nuke version this or that will be available he would just answer with a shrewd smile: “In Nuke 6.n, with n bigger than one.”

The Foundry is planning to release Nuke 6.2 this summer and expects another release at the end of this year, most likely featuring the first helping of adopted technology from Katana.

Making it faster!

Bruno Nicoletti on RIP,
originally uploaded by Phil Strahl.

This was the title of the following presentation by Bruno Nicoletti, CTO and co-fonder of The Foundry. He stressed that there has always been the problem of any software nothing is ever fast enough and before continuing made a disclaimer: “It’s going to be very technical” before he continues to get to heart of the problem’s possible solution, the GPU. “Many people are obsessed about the GPU and we are not taking quite the advantages of it as we could.”

GPUs are really good at image processing and programming them is easy. But getting peak speed is hard for programming GPUs is different, and there are lots of new and interesting technologies like CUDA or OpenCL. Simple tasks such as blur and color correction pose no problem, but graining or motion estimation is quite hard to do, even with CUDA it still is complicated.

Bruno Nicoletti,
originally uploaded by Phil Strahl.

The Foundry codes everything in C++ and less OpenGL. If any OpenGL code is needed (like for Mari) it is handcrafted and different for every application because “We want the best performance on all devices” Bruno boasts.

So what are the problems? The first problem is that GPU code isn’t as fat as it could; then everything has to work exactly the same on all devices. For basic stuff that’s easy, but producing the exact same results for two completely different code-streams (C++ and CUDA) is hard and almost impossible to manage. Manually optimizing code is a complex specialist task and works differently for every device, makes the code much less legible, is tedious and slows down delivery profoundly. Or what happens when new hardware gets out? Then every bit of code needs to be adjusted to deliver the same results, manually again, which makes the product prone to bugs and, again, the process is painful and expensive.

So the task was to produce code only once, that needed to be clear, fast, legible and easy to port. The Foundry came up with what they call “Blink“: A programmer writes abstract C++ kernels to process images. The code for a certain device can be translated from the original code automatically for various devices such as CPU or GPU, taking the advantages of either. Bruno was showing a demo of this with the Kronos re-timer, running on a GPU via CUDA. “But this is only the start”, Foundry want to continue with SSE and OpenCL and yet more as well, clever processing graphs and run-time code generation.

The upcoming versions of Nuke will also incorporate these changes over time. Since Nuke needs scanlines whereas the GPUs work with tiles these changes will be additions, because “Nuke will always be processing processing scanlines” Bruno assured the audience, “but we will be implementing this into some realtime nodes, so sections of Nuke may be redesigned to work with tiles”.

As the demo showed, the GPU is surprisingly faster than the CPU with Kronos, although it is highly dependent on the chip itself. Yet it is possible to mix GPU and CPU calls for an overall higher performance. Since motion estimation already works this way, it can be expected that this will show up sooner or later in the Furnace and Ocula plug-in sets. “And it is even possible to run it via a host application”, so I guess that’s good news for all of you After Effects and/or Final Cut users.

I Want A Pony!

Andy Lomas on Katana,
originally uploaded by Phil Strahl.

Andy Lomas continued the Foundry session talking about Katana. Since I first heard some squishy descriptions a year ago I wondered what Katana actually is. Now there’s a press release and a FAQ on it on The Foundry’s website.

As we all know, Katana was developed in-house at Sony Pictures Imageworks for the past years as heavy-duty lighting and compositing package. But that was about it. If I had seen this article on fxguide, I would have already known. I strongly suggest you should read up on it, because I won’t repeat here in my clumsy words what has already been put to hypertext already.

However, I will quickly sum it up to the lazy among you: Katana is Sony Pictures Imageworks proprietary node-based tool for lighting, compositing and distributing render-jobs. It is asset-based where you can load your assets from your favorite 3d package and manipulate every little detail, when you want to also on a per shot basis. Katana sits on top of the assets, so it’s totally non-destructive and rule-based when through it overrides are performed.

This allows the artists to start lighting a shot, when the assets are still in production, for example, and to develop a look early on. Thanks to clever versioning it is easily possible to produce a number of suggestions and to get back to the right one easily.

Sony has developed its own format for Katana, one which handles assets as assembled components in a hierarchical structure, e.g. a city is composed of various blocks, which consist of various buildings and each building has a roof and so on. Katana won’t lode the full scene graph until the artist exposes it. Only what gets expanded in the hierarchy will appear visible in the viewer.

An interesting thing is that Katana is, according to Andy’s slides, render-agnostic which means you can set up your render passes in Katana and decide there for a renderer such as Arnold or Renderman, you can even easily plug your own renderer into Katana: When your renderer understands the concept of a shader, Katana can work with it. A huge advantage of this is, that you have a consistent interface from lighting to finish, and work in one environment all the time.

Katana’s GUI,
originally uploaded by Phil Strahl.

In Katana you can choose what attributes you want to expose and/or modify to the renderer. The full assets with high-res-geometry and textures only get loaded via the renderer, so you only deal with what’s important and don’t get lost by a cluttered scene: What Katana can defer, it will defer. Also in Katana you can trace why something is rendered the way it is, like where an asset gets its specular map from and so on. For example, a Ds tag means that the value is the default from a shader, whereas Ls indicated that the value was set locally from a node within Katana, so an artist can always “debug” shading and lighting issues.

If you want to get your hands dirty with Katana yourself, you can: There’s an API for your own company’s C++ plug-ins and, of course, Python support. The scripting language in Katana itself is CEL which can be used to select certain nodes, e.g. to apply a certain material to all geometry nodes that have a certain name match, certain tags or attribute matches but also by a collections. This allows to override parts that do not exist at the time but will get modified once they are ready. There are a lot of further options to use CEL to override multiple object, like only turning on the specularity of a whole scene — bang! — you just made a specularity pass.

Now let’s have a closer look on the user interface and the workflow with Katana. Andy showed the audience what instantly sold it to me, namely the “I Want A Pony” menu option, which creates a pony-shaped node in the graph view. “This is actually one of the horses of Beowulf, so it’s technically not a pony. But we use it as a primitive here in Katana. And it might even succeed the Utah Teapot” he added jokingly. The further now one drills down on the asset, the more parameters get passed on to Katana until you arrive at the vertex-level. In theory you could make whole 3d-animations and models in Katana.

Compositions look like trees with roots: On the top you load (and/or) group your assets and materials (like branches and leaves), assign nodes that perform certain changes and in the end you have multiple renderers and render-passes (like roots). And the interface is, of course, very customizable and panels can be docked or torn off.

Materials can be stacked together into one stack which has only on input and output connection to the outside to keep the complexity low where it is not needed. Everything can be inherited to child nodes, even transformations, if you want to. So for example making a wet material can be just done by creating a child node that inherits everything from its parent but diffuse and specular values.

The gaffer-node is a “one-stop-shop-node”, as Andy put it, to execute a macro and to create lights with the most common attributes already exposed.

Katana composition can be references or, by using the KatanaSdtBake-node, be baked for other artists to use, have live groups in a macro or share real Katana scene graphs (i.e. the compositing-scripts) via a library.

In the end, Andy announced that Katana won’t probably ever ship as a single product by The Foundry. If it ever will, then probably with its 2D capabilities stripped out of it which will in turn be used in a new Nuke version. Katana features in Nuke will probably show up still in 2010. There is potential in Katana as a re-lighting engine, “but that’s quite far down the line at the moment” Andy concluded.

35 Years of Slapping your own back

Lynwen Brennan,
originally uploaded by Phil Strahl.

Lynwen Brennan’s lecture on ILM‘s past 35 years was as mundane as it was boring, but it was no surprise hearing that from the President and General Manager at Industrial Light & Magic. At least her presentation had many pictures in it (in the 70′s, every guy had an impressive beard), tables and schematics of why ILM is the most successful company there is; that ILM invented digital editing; that John Knoll and his brother came up with Photoshop; that the first shot featuring digital compositing (and not a mere optical one) was in The Abyss — so it was a nice blend between appearing all corporate and fun facts. Lynwen was reading her hour-long presentation that left no questions open, partly because of the fact that it contained nothing that you wouldn’t find on Wikipedia. The primary message was that ILM tries to raise the bar, keeps the costs in check and wants to define new standards.

ILM panel discussion,
originally uploaded by Phil Strahl.

What followed was a panel discussion about the global production at ILM, since ILM has also a studio in Singapore (that’s where the Clone Wars series gets produced, including the Nintendo DS game, see last year’s fmx coverage), and works closely with other post-houses such as Pixomondo.

My lack of journalistic skill now really shows because I didn’t get the name of the moderator — my apologies!

The panelists were Dennis Cooper, Lucasfilm’s Director of Global External Production; Gretchen Libby, executive in charge of external production at ILM; Mohen Leo, ILM’s Singapore Studio Supervisor and Thilo Kuttner, CEO of Pixomondo whose biggest contribution to the discussion was the fact that he held his microphone like an umbrella all the time and appeared overall subsequently rather faint (see photo below).

The first question asked to the panelists was a basic one: Why global? In ILM’s opinion it’s a good way to keep the cost down and to maintain quality by having many options. For Pixomondo (who have offices in Babelsberg, Frankfurt, Munich, Stuttgart, London, Shanghai and most recently Los Angeles¹) being global is not only to leverage the global talent but also a way to keep everybody busy.

Since communication is one of the strongest factors in collaboration it neither ends for Pixomondo nor for ILM at simple email exchange. Thilo strongly proposed also understanding the different mindsets and cultures instead of forcing your own ways upon your collaborators. To ILM it is important that every bit has to be as clear as possible, “It’s not only about being clear in what you say, it’s also considering what the other one hears”. Mohen further stated that for the artists in Singapore it is important to have direct access to their respective supervisors in the US. “The better you know a person, the better you know what he or she means when saying something. The simple act of having lunch together can help a great deal in that respect” Gretchen Libby added.
The diverging mentalities really do put international communication to a test, in Singapore, for example, direct yes or no answers are avoided so different strategies needed to be sought. “It’s important to know that when you ask ‘Can you do the shots till Monday’ and you get ‘We do everything we can to have them on Monday’ it actually means ‘No’.”
Pixomondo even introduced a command-list to get the most urgent and most important communication dead-right, independently of the culture. “At ILM we know we are all in a very visual field, so we draw a lot of pictures.”

Thilo Kuttner,
originally uploaded by Phil Strahl.

To ensure a continuity in production ILM is a big proponent of creative leadership and development of long-term relationships. “You don’t produce where it’s cheapest, you go where you get the best ratio between quality and price.” In terms of production pipelines Pixomondo orient themselves on ILM, who are really fond of their proprietary stuff, yet they insist that they are open to change.

The process of going global (or not) starts very early on for ILM, when they look through the script. Based on this script they assign different tasks to different studios all over the world, studios with strengths in certain areas, although “we look for synergies as well.” And ILM goes “where we can find capability and punctuality”, both equally important in any big production.

“But is working together with other studios not like training your competitors?” the moderator asked with a wink. But the folks of ILM stayed relaxed. “Talent and competition are both everywhere [..] we rather look for people who want to work for us”, to them everything is about building long-term relationships with their partners. “Also our clients want and need to know where their material is being worked on”. For Pixomondo working with ILM was also a long-term strategy as Thilo added, “not least because it’s an honor to work with ILM.”

In the VFX business (especially with big players such as ILM) security is a hot topic. The process of evaluating the security standards of a potential partner begins very early on, “basically with the first phone call”. They need to fill out a questionnaire on who has access to their premises, to their data, what kind of passwords they use and so on. “And site visits. Lots and lots.” The material they get to work on is watermarked, data transfer is password protected and so on.

So what about the future? Thilo concluded that the business will continue to go more global (big surprise), and also ILM only wanted a “continuation of where we are now”, perhaps with a 24-hour feedback cycle; an attempt to work with the time-zones instead of against them.

The usual colors

Tim Sarnoff,
originally uploaded by Phil Strahl.

The visitors in the König-Karl-Halle had thinned out a bit for the upcoming lecture by Technicolor’s Tim Sarnoff on “Working Across the Globe”. Maybe I was a bit tired by the previous presentations or the lecture really lacked much structure and in the end there were even less people in the audience. In the end it was just a company presentation of Technicolor with the occasional buzzwords thrown in, mixed with some commonplace information such as “You remove a certain degree of risk when going global; One has an unlimited talent-base when being global; Being a global company needs work and consciousness” and from the Technicolor promo video I noted down “It’s a business of passion enabled by technology”. I was glad when it was over.

On the plus side I swapped business cards with a German screen writer who had her degree from the prestigious Filmakademie Baden-Württemberg and for lack of job opportunities in Germany (“I hate German drama!”) she resigned to writing a novel for the moment. Networking on the fmx was as easy as always.

Down the rabbit hole

David Cohen,
originally uploaded by Phil Strahl.

The last lecture of Tuesday was by Ken Ralson and David Schaub from Sony Pictures Imageworks (SPI) and all about Tim Burton’s Alice in Wonderland in which they lined out some of the production process. “First thing we learned was that Tim [Burton] hates storyboards. So we started with gathering a load of reference material with our starting point being the 1922 Alice in Wonderland movie. In theory they wanted to establish a style and then shoot which in practice didn’t really work out, in the end changes needed to be made until the last minute.

For the pre-production they found an artist at the portfolio section of the CG society² they liked and gave him a little piece as a test. They liked the result and hired the guy from Germany. “I bet the CG society will get a lot of new members as soon as this lecture is over” I thought to myself.
The concept artists painted a lot of designs and suggestions until Tim Burton would eventually say “This feels right” and the artist could ponder about what it was that felt right to Tim and develop it.

What SPI did in the beginning was also a test of enlarging Johnny Depp’s eyes in a shot from Burton’s great film Ed Wood. The result was hilarious and “when we showed it to Tim and Jonny they both cracked up so we knew it would work. [...] and after a while working with it, Johnny’s real eyes seem always a bit too small.”

The principal green-screen shoot was incredibly tight so everything needed to be planned and considered in advance. “It wasn’t an easy task considering there was so much that needed to be CG, like Crispin Glover whose whole body needed to be replaced, or Alice changing between three different sizes throughout the film”. All shots with extras or supporting characters were shot separately.

Since the set was, apart from the occasional set pieces, totally green, the team of SPI even installed a pre-vis system on set for Tim Burton to watch. “He looked at it, then grabbed the monitor and turned it away. Tim preferred looking at the raw greenscreen play-out³ but that’s okay.”.

The raw footage from the greenscreen stage they presented looked, truth be told, incredibly silly: Mia Wasikowska running on a green treadmill for her life or riding on a green upper half of the Bandersnatch that was shaken by some grips to a click-track; Johnny Depp standing heroically in a crazy costume with weird makeup; Anne Hathaway in an extravagant costume riding on a green vaulting horse carried by three guys acting all horsey in green; there was Matt Lucas plus stand-in in a green pear-shaped costume with tracking markers all over as the Tweedles; Crispin Glover in green shoulder-pads every quarterback would kill for walking on stilts, trying to act all normal while two guys in green were always walking next to him, ever alert to catch Crispin in case he would trip. Yes, it looks absolutely retarded. If it wasn’t a multi-million dollar production you would just laugh, then weep and then facepalm.

In order to realize the big head of the Red Queen, the shots with Helena Bonham Carter where were shot in 4k resolution and everything but her head scaled down 50%. In the scene introducing the Red Queen she wipes a drop of jam from one of the frogs’ faces and sticks the finger in her mouth. To realize this interaction her hand needed to be painted out of the plate with her head, a terribly challenging piece of paintwork. And thanks to my contributions to Ninja Assassin I know what I am talking about.
Another demanding paintwork was Stayne’s hair: All the concept paintings of him showed a knight with big shoulder pads, so for the greenscreen shoot Crispin Glover was put in a green costume with said shoulder pads, yet in the end Tim Burton made up his mind and went for an armor without shoulder pads. Needless to say that the missing hair needed to be painted back into the shots.

Another thing about Stayne was his height. As stated above, Crispin was walking on stilts on the set but capturing his movements on set resulted in an awkward animation and looked much like it looked on set: Like a guy walking on stilts. So the animators resorted to good old manual keyframe animation for Stayne while keeping the essence of the movement on set for him. This reminded me of a quote from Pete Travers on The Making of Dr. Manhattan at last year’s fmx: “If you have developed the best tracking system in the world but it hinders the actors you end up making perfect tracks of bad performances. Which are totally useless”.

The appearance of the Cheshire Cat was easier in contrast. The animators started out with a very cat-like animation but Tim dialed them down until the cat was almost not moving at all. And to achieve the terribly wide grin, the cat’s jaws needed to transform as well.

Since the feature was in stereo ⁴ (and what isn’t nowadays anymore?) but shot with only one camera, the stereo-conversion of the live-action footage was done in post, by rotoscoping and/or projecting the live-action cards on 3d-geometry within the scene.

After this refreshingly interesting and engaging lecture it was also this year the time of Shelly’s Eye Candy Show.

Sweets for your eyes

Also this year Shelly Page from Dreamworks Animation assembled 50 minutes of animations she considered highly worth watching. Some would also get screen during the International Festival of Animated Film I also attended after the fmx was over, but more on that later.

The screened films this year were:

• Mobile by Verena Fels, a cute little animation about stuffed animals hanging on a mobile with a funny pacing and rewarding pay-off in the end.
• Stylo, animation by Passion Pictures, the Gorillaz music clip featuring an amusingly smug Bruce Willis and artful integration of cartoon characters in live-action footage. “Stylo is directed by Jamie Hewlett and produced by Cara Speller for Zombie Flesh Eaters, with live action through HSI Productions in Los Angeles, and animation by Passion Pictures in London”⁵
• Log Jam Series by Alexey Alexeev, the first clip formerly known as KJFG No5 ⁶. This hilarious little animation was also in Shelly’s reel last year but a producer urged Alexey to make make more, for a series. So he continued with some more, all equally pointless yet entertaining. Be sure to watch them all, they are called “The Rain”, “The Snake”, “The Moon” and “The Log”.
• Anchored by Lindsey Olivares, her senior thesis film made at Ringling College of Art and Design after Romans 15:13. I enjoyed the style of flowing watercolors much, although the topic wasn’t so much my thing.
• Inka Bola by students at Gobelins, an entertaining piece about a spoiled toddler and his guard. The animation is superb (but frankly I don’t expect anything else from Gobelins) and the whole piece has the speed and style of a Disney or Dreamworks short. Very enjoyable.
• Evian Babies animation by MPC, a weird trip to the outskirts of the Uncanny Valley: “Michael Gracey has directed Evian’s latest commercial Skating Babies a multi-national campaign bringing together choreographed roller-skating babies and the re-mixed street sound of The Sugar Hill Gang’s Rapper’s Delight. Created by the agency BETC Euro RSCG, the spot was produced by Fabrice Brovelli, Head of TV at BETC and Jaques Etienne Stein at Partizan. MPC created fully CG baby bodies and carried out extensive live action head replacement and compositing as well as large scale digital matte paintings to extend the park environment for the TV and online campaigns.”⁷
• 5alive Dodo animation by Passion Pictures, a TV commercial with funny character animation of a dodo dancing to I’m Alive by Don Fardon.
• LowLow Cheese animation by MPC, another TV commercial with a photo-real mouse avoiding a crapload of mouse traps. Yes, we’ve come a long way from Stuart Little.
• RockBand Beatles intro by Passion Pictures, also this year the RockBand series has yet another stunning intro to worship. In just two and a half minutes the intro tells the story of The Beatles’ success in something I just call “masterfully art-directed pictures” (and sound!) and is highly enjoyable to watch. I even found a short behind-the-scenes talk about it on YouTube!
• The Little Boy and the Beast by Studio Soi. Oh, I just love that one, it’s about a boy with a depressed mother and how he deals with the situation. This is not only a tough topic for children, it is also skillfully executed and designed, from the plot to the final playout. It also ran on the children’s section at the International Festival of Animated Film and the kids loved it. To me it was the best short on Shelly’s reel.
• Lost and Found by Studio AKA, a 25-minute short of a boy confronted with the sudden friendship of a penguin. It was nicely done, though there was a lot that bothered me such as the narrator off-screen that not even commented what was going on but only repeated what the pictures showed; the sequence out on the rough sea which lasted waaay too long and the water itself that was too photo-realistic to fit convincingly into the style. It was nice, yes, but about ten minutes too long and nothing spectacular — sorry.

Phew! What a day! You see now why it took me so long to get this from my head to my blog, there was just so much knowledge to chew and digest. I hope you stay tuned for the upcoming reports of the following days. Hopefully not as lengthy, though.

Feel free to check it out yourself here, where you can also see the sneak peek of it. Now I gotta get some some sleep, just came back from holding a live tutorial on the FH Salzburg. Exhausting, but fun!

There’s quite a bit I have planned for in 2010. First and foremost I’ll get busy on some new AEtuts+ video tutorials introducing Nuke to the After Effectors among you all and something I have in mind for quite a while now, working title: “The Art and Science of Rotoscoping”. (Everything’s “art” and “science” in visual effects, if you read the books.)

There’s a new series for flickr that I have in mind by the name of “Austrian Details”. There are so many things I encounter that are so typically Austrian in some way and that people from other places probably wonder about when they see it. When it launches I’ll tell you here.

I’ll also be holding Tutorials on the Salzburg University of Applies Sciences again, this year with a main focus on compositing, color grading and VFX.

In May the fmx/10 in Stuttgart will be held again and yours truly is looking forward to cover the whole event on this blog like the years before for those of you who can’t attend.

So I’m getting all educational it seems. And creative. I’ll try to post somewhere something creative I did every week. Since I’m an aggressively creative person that shouldn’t pose any problems to me.

Life’s incredible again! Stay productive and wise! Any single ladies out there, btw?

Pipelines of War

Greg Mitchell ,
originally uploaded by Phil Strahl.

The day started even louder than the fat guy jumping down the stairs above my room at 6:45am: With Gears of War 2 (GoW2) and how Epic Games thought up streamlined their production pipeline for those. Greg Mitchell a big guy, well presenter and Cinematics Director at Epic worked twelve years in television before he switched gears (pun intended) and went into the game industry. He already worked on the cinematics of the first Gears of War (GoW) but wasn’t quite 100% happy with the outcome: Not all was motion captured and so sometimes the animation data had to be sped up or slowed down; e.g. a character walks with 70% speed of the captured motion but talks normal, it just looks weird.
So Epic Greg set himself the task of making everything better than in GoW, to stick to a consistent filmic ¹ style and look.
In GoW there was no pipeline, everything was done by the game artists more or less parallel to their tasks within the game. It worked, yes, but it could have been much better.

Greg laid out first of what the needed pipeline need to consist of in order to determine the scene scope and the needed assets. In the beginning there are audio plays (“radio plays”) and animatics for each scene. While art and level assets are being created the mo-cap recording starts with constant input from set- and level designers (e.g. with blocking diagrams) to give director and actors information about the environment the characters are in. On GoW2 Greg worked with real actors instead of having people “to pull away from their desks”, made enough rehersals before the capture and played back the edited soundtrack with the voice actors on set. All that led to a much higher and better quality and the production speed improved significantly.

After the recording the layouting process starts where the scene takes shape, gets a pace and the cameras are set. Although one might think that motion-captured data is pretty rigid to work with it is not. To get better angles for over-the-shoulder-shots the characters can change their positions a bit or some parts of the animation can be repeated between the shots, e.g. to use a walking sequence twice in succession to give the impression of a greater distance.

This reminds me of what I’ve learned from my mentor Fraser McLean who always told us that we can cheat if it helps the story. He’s currently working on a book about the history and role of layout in traditional animation vs. computer animation today. Because layouting is the most important step in every production I think you should get it once it’s out.

Once the layout gets locked, lighting and effects artists add atmosphere and mood to the scene while the audio department populates the soundtrack with effects and music that is specifically composed for key scenes. You need to bring in the game-designer(s) and producer(s) into the feedback loop as early as possible to keep the revisions to a minimum. Needless to say that there is a constant bugfixing and polishing going on; “With cinematics you’re never done. Never. But at one point you just have to say that it’s finished.” Greg ended.

The art behind making a game

Matt Aldrich,
originally uploaded by Phil Strahl.

Matt Aldrich, the Art Director of Lucasfilm Animation in Singapore, shot up a big image of the package design of the game he was working on. Star Wars Clone Wars for the Nintendo DS. It is sacrilegious to say that you’re sick of Star Wars? I just know that I really am. Nevertheless I tried to be as unbiased and open as possible. First off Matt showed excerpts from the design document which was really thorough and had everything plotted out very detailed. The level design was then outlined in Google SketchUp to visualize not only the key areas but also the players progression through a level. After approval of this very rough layout the art process is started: Based on the detail-lacking SketchUp renderings storyboards are drawn from the player’s perspective int he level and the key moments he or she’ll experience. This is mainly to point out issues before it is costly in time and hence money to make the necessary changes. So take your storyboards and discuss them with every member of the team. The engineering-guys for example will be interested in the amount of polygons simultaneously on screen, the character designers in how close we see the enemies and so on.

The next step was defining a color arc of the scenes in the level. In the level Matt was showing there was a progression from cold, steel-blue colors in the beginning to hot, orange colors at the level-boss fight scene. While it is possible to spend quite a long time drawing those, it is much faster to find reference images and color correct them in Photoshop or do quick paint overs until the colors are final. The focus also laid on key areas for these pictures because nobody has neither time nor manpower to have detailed concept art and paintings for all areas of a game. Once camera angle and perspective where also laid out and locked, concept paintings could bring in the color from the references and add details. In fact those images where so big, that they also became a source for textures for the game. The concept paintings also showed whether it was for the player easy to progress quickly enough through the level. And again, concept art is there to open the discussion and to make it easier to be specific: “The pylon on the left should point in the direction of the player’s goal” is much better than “give it a slant and make it look good”.

At any point it is important to always recall the limitations of the target system. The NDS has very small screens (256×192 pixels), the texture memory is very limited (so the use of vertex lighting was quite important and extensive) and it is a device you can carry around and play in every light situation. So the art department had to focus on good contrasts, very legible silhouettes and a clear level design.

It is incredible what those guys in Singapore did on the DS: The characters for the in-game cinematics have a quite sophisticated animation rig, so they can show facial expressions and talk in lip-sync. I was shocked and awed. But in a good way. Matt went on with how they expanded the Star Wars universe and developed parallel to the TV show for new planets and space ships. But let’s be honest guys: It looks pretty much like any fantastic sci-fi stuff, like all Orcs and Elves and Goblins look alike throughout the fantasy-genre. So I didn’t take any more notes in this presentation. I only know that I want to give some DS homebrew stuff a chance.

Nuke ‘em

I switched rooms and went to the heavily crowded lecture ambiguously titled “Stereo-3D Film Post Tools and Algorithms which turned out to be a presentation of what’s hot and steamy and in beta in Nuke 5.2 by at The Foundry. Surprisingly I got a seat in the second row and had a good view on Simon Robinson’s presentation. In fact it was all about fixing terribly shot stereo ². Simon, head of development at The Foundry, really knew what he was talking about and showed all tricks in Nuke rather than just running a PowerPoint visual hell presentation.

First of all he outlined how to work in Nuke with stereo imagery. You either can use the JoinView node on top of your tree after reading the different eyes ³ or have them already combined in a single EXR file. I asked Simon afterward which one was faster, but he said that it would depend on where you read your EXRs from, what type of CPU you use etc. So I make a wild guess and say that there’s practically no difference. Simon went on to tackle specific stereo problems that can occur in live-action shoots. “If everything was shot right in the first place, none of us would be in this room.” Well spoken.

• The O_VerticalAligner node can compensate for incorrect image alignment but obviously can’t deal if there’s a shift in parallaxes because of it.
• O_ColorMatch is another node that helps to match the images of the stereo-cameras together. Color discrepancies often occur when one eye was shot through a mirror in order to get a closer interocular distance ⁴. While this node does not a perfect job, it does a rather well job and makes it a lot better.
• Nuke can calculate a disparity ⁵-map from the two eyes via O_DisparityGenerator. The stringer the color, the stronger the disparity is. Currently this flickered a lot but “see me in a presentation in a couple of months and this will be much better”. They’re always improving.
• There’s also the possibility of setting a convergence point. The advantage is, that it can be done for any pixel in the image, so dragging a convergence point over a moving object can keep the focus on it.

At this point Simon switched to the anaglyph view in Nuke and I got a little upset. When entering, people were given a set of paper-polarization filters that have a distinct gray color ⁶. So on the appearance of a blue/red anagylph image on screen about 95% percent of the people I saw around me put on their filters, without even thinking about it. You just won’t see a 3d image in an blue/red anaglyph-image when wearing pol-filters on your nose. Some folks kept them on for as long as 30 seconds before realizing it. Sheesh! And here comes the kicker: Most of these people did it a second time with the next anaglyph image just a couple of minutes later. Some people just drive me nuts!

• A clever and time saving idea is the ReConverge node that pushes everything the artists did from one eye to the other, e.g. roto or paint. “It won’t match perfectly, still puts you more than half the way through. You only have to tweak it instead of recreate it.”
• O_InterocularShifter comes in handy when the interocular separation between the two eyes was shot too wide and you have to fix it. This node calculates a new set of stereo-cameras that are positioned between the original ones. Currently it took Simon’s notebook about 20 seconds to calculate a frame. “It’ll be faster next time” he promised. Still it is nothing to correct an entire movie with because there will be occluded objects in the right eye and occluded object in the left eye which can’t be magically thought up by the software, the disparity estimation won’t work then. So it’s more a tool of last means rather than a way of remastering your stereo IMAX movie for television. However it can assist CG pipelines.

So does stereo just keep making our lives harder? Yes, still some things work better. Like camera tracking because you have way more depth information which in turn results in a much more stable disparity map. from that you can pull a Z-map of your scene and add things like volumetric fog in post-production or correctly pulling the digital lens for some depth-of-field-effects. Another thing that will be coming along is that more and more metadata from the shoot will be used in the compositing process, eventually even autmated. Until now we had the pleasure of running around with clipboards, tape measures and constantly bugged the DOPs. At least I know I had.

What’s still in development are things like lens-distortions or how to deal with optical effects such as lens flares or blooms in stereo. What I have learned from last year is that you either have them in either both eyes or no eye. Further Simon talked a little about using more than two cameras to get even more information form a live scene, “The algorithms are there.”

Essentially this lecture was was a down-to-earth showcasing of The Foundry’s Ocula Plug-In set. If you want to bug the poor man even some more: Here’s his address sam@thefoundry.com.

Small is Beautiful

Richard Hilleman ,
originally uploaded by Phil Strahl.

After the break gaming veteran Richard Hilleman from Electronic Arts held an inspiring lecture about the evolution in games. We all knew that in the early 1980s pretty much everybody with a computer, programming skills and a good idea could make a game and, eventually a lot of money. Fast forward 25 years: Today there are a handful of big players and about 50 teams (worldwide) that can pull off an AAA high-def game costing 25 million dollars ⁷. While all these big studies can’t afford to take any risks and have to please a very broad audience, the small designer can do whatever she or he wants because the stakes a lot lower. If four people get together, work on a browser game for a couple of months that did cost them, say $500 in total, and they make a profit of $10,000 that’s a huge profit margin. Yet $10,000 wouldn’t probably even cover EA’s monthly coffee bill.

So how do you make a great product then? It has much to do with yourself:

• Show passion not only in making the game but also for its content. Richard Hilleman has a passion for Football. So he created the first Madden NHL which was, as we all know, a success.
• Be versatile! Nobody’s going to hire the 8th shader engineer. But if you are a shader engineer who knows how to manage a group of people, about their tasks and see the bigger picture, your chances will improve drastically.
• So learn more than your base skill and get technical as well as leadership experience. You’ll learn much more when you have to lead a team that’s so big that you can’t do what’s missing in the end yourself.
• Be curious. Explore. Obtain knowledge. “Don’t accept the box they try to put you in”.
• Learn about money and how it works with your product. From start to finish. Internalize it. Understand the economics of your product. There’s just no way around it.
• Learn people. Because “Everything you learn technically will be gone in 7 to 10 years”.
  • People are your customers.
  • People are your team mates.
  • People are your means of expression.
  • People are you inspiration.

  Change people’s minds. Surprise them. Take them on a journey. Entertain them!

You won’t need a huge target group. The target group for games usually is between 14 and 20 years old and male. They have the time, their parents have the money. But you can’t experiment much inside that target group. On the other hand there’s Pogo where the average person plays for about 20 hours per week. “Would they consider themselves as gamers? No.”. This market for casual games is evolving. There still will be the audience for high-def games but don’t forget the long tail. Pogo’s average customer are 49yr old women, for example.

“Who do you rather want to be: Michael Bay who gets $250 million to shoot some producer’s movie or Robert Rodriguez with $5,000 shooting his own movie?”

The Tale of Framestore

Andy Lomas of Framestore CFC told the tale of an English post-production house that was famous for its commercials when it set itself the task of creating an animated feature film, namely The Tale of Derspereaux. The most interesting part of it was the way they mimicked the lighting on paintings from the old masters Bruegel, Vermeer but also Bosch; that they had to use mouse-scale cameras for the proper depth-of-field effects, used filmic dollies and technocranes and made the image deliberately imperfect by blocking the view or some jitter here and there, have even more flaring and blooming and so on. Nothing new, in fact. I’ll cover the cinematography of WALL·E below.
Personally this lecture didn’t intrigue me much. Yes, Framestore showed that they can pull off making a full CG movie in Europe by themselves but there was nothing striking to me. In my opinion even the look wasn’t that top notch but still waaay better than that horrible Back to Gaya.
Andy showed their tools such as asset and production data management and so on. He talked about the shift from an asset based workflow (“what stuff is needed?”) to a shot based workshot (“what’s the story here?”). Also Andy stressed the importance of layout and previz (nothing new, huh?) as means of a creative hub, bringing the costs under control and to lock down as much as possible as early as possible. Again, bring in the clients as early as possible in the feedback loop for their involvement is essential. In fact is the final feature nothing more but a very refined version of the layout.
In the end he showed some production tools Framestore had used such as their production asset management tool Shotgun or Pick a Prop that linked the Object ID pass in an EXR to the asset database and displays the name of the prop the pointer is hovering over. This was mainly to ensure a clear communication such as “What’s wrong with the shadow of cr_p_wooden_barrel_v54″ as opposed to “What’s wrong with the shadow of that brown thing in the background?”.

Let there be light

Danielle Feinberg,
originally uploaded by Phil Strahl.

What followed was the lecture most of us were waiting for the whole week (the fmx folks really do save the best for last). Danielle Feinberg, who was one of the Pixar DOPs on Wall·E, explained how and why the feature got its distinctive look. As I already knew from last year’s Pixar presentation they really like to research extensively and try things out for themselves. And for Wall·E they found that their lighting model and camera code was outdated. Director Andrew Stanton’s vision was to show the abandoned earth through the lens of an 1970′s science-fiction-feature camera, with all the distortions and funny stuff going on. To test things out they filmed in the atrium live-scale models of Wall·E and Eve with test patterns all around and a grid on the floor on 70mm stock and with anamorphic lenses. Hey, they even got Dennis Muren to show them the ropes! So according to their tests their camera-code was adjusted. Also, they set themselves the limitations of having only a certain set of lenses (see box). Now Pixar operates a fully functional virtual 70mm camera with anamorphic lenses and all the artifacts that they bring (optical breathing, barrel distortions, lens flares with blue streaks, elliptic highlights and so on). If you don’t overdo it you get yourself a look.

To develop a look the folks at Pixar also researched extensively and came down to that 1970′s science-fiction-feature look. Orange, documentary, existing light is used and, just like Sharon Callahan said last year about Ratatouille, don’t be afraid of the dark i.e. let things go to complete darkness if it is justified. But again, Pixar failed on that. I guess they tend in general to over-light their features in some respect.

For the shading they came up with a new illumination model of energy conservation ⁸ that essentially comes down to three knobs: reflection, specularity and roughness. Basically the rougher a surface gets the less it reflects and the amount of reflected light is never higher than the light the surface received. The new shaders also are capable of ‘hot reflections’ and perform realistic fesnel falloffs themselves in the rendering.

The shading of Eve was much more complicated than anticipated because she is made up so many parts that should fit together seamlessly, yet has circuits and light on the inside and goes through quite a lot of transformations. On the other hand Wall·E’s eyes were also an important part for his performance so he wouldn’t look dead (too reflective eyes) or creepy (too little reflecting eyes). He got his final appearance by lighting the aperture blades inside so they would break out visually from the blackness of his eyes.

Because I was more concerned with the technical side of this lecture I don’t have any notes taken on the other topics that were touched, but I bet there’s an artbook already out where you see many of the beautiful drawings, silhouette and color studies and so on. Pixar artbooks are either way and obligatory possession and resource, even if you’re only on the outer rims of the industry.

Waltz with Michael

Michael Faust,
originally uploaded by Phil Strahl.

The lecture was just called “Waltz with Bashir” and the only thing I knew about the film was a 5-second clip I had seen many months ago which made me eager to see it. Unfortunately I missed seeing the film once again. But not this lecture featuring the stunning look the Bridgit Folman Gilm Gang hat achieved. With Adobe Flash! Michael explained that it was the first feature film of The Gang, nobody had worked on something that big before. In the course of the pitching they animated a scene using the cut-out technique in Flash by separating body parts and just moving them around until a new keyframe was needed to be drawn. At first the segments were rather large and soon they figured out that it wouldn’t look good enough. So more and more shapes were broken into their components until a face was nothing more than a flesh-colored blob with dozens of tiny black snippets that the animator used to animate the face.

The backgrounds often came from photo references that had been traced and painted over in Photoshop; some elements were completely thought up and yet they integrated perfectly into the realistic environment. Michael, who worked as an Illustrator and did some backgrounds said that it was hard for him at first to change his style he was used to from his oil painting to something so completely different.

When the animation work on the film could begin, the director already had the film finished, leaving out lots of black holes and studio reinactments of what needed to be animated in the final film. So the layout phase began. It was done mostly traditional and very sketchy with indicators of what needed to be a new keyframe that had to be drawn. The characters and poses were drawn by hand and then in Flash painted over, the backgrounds traced in Photoshop to match the very narrow color palette. Any effects had been done in After Effects such as trails of smoke. Michael brought some animatics for us to view (it still was odd watching him open .swf files of what ended up on real film stock) and they looked pretty much like uncleaned finals, their quality was just outstanding. Like the rest of the film. Because of the tight budget there was no room for motion capturing or painting every frame by hand. So this Flash-based cut-out technique, as tedious as it may seem, was still faster and cheaper than traditional animation.

What party?

There it was again: My lack of sleep kicked in hard and so I decided against watching the animations from the SIGGRAPH Asia which I regret bitterly. I went back to the hotel, slept some hours and woke up just in time to visit the closing party. But you know me: I don’t like parties because there’s nothing for me to enjoy: People are drunk and pushy, music is too loud to converse properly (also there’s not much to discuss with drunks) and the only people you meet are party people. So I stood in the hotel and tried to catch up some sleep for the journey home. I failed.

After enjoying the breakfast a little too long I was rushing down Königsstraße in my car so I would make it to Pixar’s RenderMan presentation. I already knew what it was going to be considering last year (“The Über-Sprite”, the rocket, the fast-rendering motion blur) but Pixar is rather generous in handing out posters and presents and I wanted me to get another teapot for my collection ¹. I was too late, the room bursting with people. Obviously, word had spread that you get presents. People can be so greedy. I asked if I could make a reservation for the afternoon but it was in vain.

There I was standing, lacking a teapot and a clue of what I wanted to see instead. I headed to the biggest hall and ended up in “PhotoReal Facial Animation” by Patrick Davenport and Steve Caulkin of Image Metrics. They showed the sample clips I already knew so it was no surprise to me that… (click “show” to view spoiler) [spoiler]…Emily’s head was CG.[/spoiler]

You can find the clips also at YouTube if you’re interested.

The crazy stuff Imagemetrics does is providing face tracking with only the use of a video camera. The tracked regions of the face are then moved on a CG model. Tweak the keyframes and you’re done. It’s that easy! Well, it’s not. Steve Caulkin laid out the long way to their Emily demo which occurred to me as not really time saving: Apart from photographing the actress’s face for the texture, there also had to make a cast of her teeth but the molded teeth wouldn’t necessarily fit correctly so you end up taking x-rays to learn how to place the teeth correctly. And that’s only the beginning.
When scanning the different expressions of the actress the data was anything but coherent so somebody had to clean up all the meshes (about 55) and get the details out: Pores and such can only be done with a bump or displacement map. It would be just too much for the statistics-based tracking algorithm.

Steve Caulkin owes me a venti Caramel Macchiato. His presentation was in-depth and very interesting but, alas, Steve is more a guy you put in front of a C++ compiler than in front of an audience and it was hard to follow his low pace.

So I ended up at Starbucks with an iced caramel macchiato before making another attempt at getting into one of Pixar’s presentations. I queued up 20 mins and before they opened the doors there was already not much oxygen left. And I felt the urge for another caramel macchiato.

Pixar’s Carreer Gears was a again a valuable information on how to apply and how to put your reel together for Pixar. Right in the beginning the panelists ² asked the audience to raise their hands of what position at Pixar they’re interested in. To sum things up: Two thirds were character animators, many wanted to become story artists and only a few people were interested in the other stuff. And I bet I was the only compositor in the whole room. Here’s what’s strange: Last year I was told that Pixar doesn’t really do compositing which I thought was a joke or they wanted to pull my leg. Today they also didn’t say anything about job openings or positions in compositing. Very strange.

The panelists talked about their experiences at Pixar and how they got their job and spread the usual tales of people who were hired right off the college. Then they took questions. I must have dozed off somewhere in between but it was mostly asked on the process of applying and what Pixar is looking for. Here’s the stuff I remember:

• Don’t send in a reel when you have nothing to show.
• Put your name on everything.
• Have the DVD region-code free and tested to play on a standard set-top DVD player (NTSC and PAL both are fine).
• Apply for a certain job instead of just applying for the database.
• Send every 8 to 12 months an updated reel to show how you progressed.
• Don’t send every week new reels.
• Write a decent cover letter. They’ll read them.
• Don’t forget the all-important shot-breakdown. Preferably even on screen.
• Don’t chase the ostriches on the front lawn (I guess that’s where I dozed off).

Interestingly I gradually lost my interest and my caffeine addiction kicked in hard after an hour so I left for a chili dog and a precious cup of coffee. At Starbucks they either love me or hate me.

For lack of motivation to look for the right screening room for “Analog Artifacts in CGI” I went with the crowd to witness “Skin & Lighting Research” by Christophe Héry of ILM whom I already know from last year.

Holy moly! In his presentation I saw more formulas than in my whole college education ³ In fact he told nothing new about subdermal distribution and the models on how to calculate them (although I didn’t understand much of the math). So far, so good. But what If you can’t afford raytracing because, say, your artists produced more vertices than the final rendering will have pixels (see Pirates of the Caribbean II: Dead Man’s Chest)? You’ll have to have an point cloud based approach to dodge memory demanding raytracing. And when you don’t have raytracing going on RenderMan really does the trick fast and good.

Cute as a Button

That lecture served as the perfect introduction to what we all have been waiting for: The Curious Case of Photoreal Head Replacement.

Jonathan Litt had a huge presentation explaining the lighting, rendering and compositing of that huge task. How do you start? They started with a artfully crafted latex-maquette of Brad Pitt’s face made old, for it had a really realistic appeal in subsurface scattering and served as most valuable reference when comparing renderings of the CG head to it.

The head itself was done in Mudbox (yay!) and in it’s highest resolution had about 4.5 million polygons. This high level of detail was preserved by using displacement maps, that further were driven by curves so wrinkles would get stronger or weaker depending on the facial expression. The eyes were modeled and textured anatomically correct (I’ll just throw some expressions at you of what they considered: caruncle, meniscus, conjunctiva, sclera, cornea). As further reference they had a extreme-high-res photograph of Brad Bitt that you could see the micro-wrinkles between his pores. “That’s thousand dollar pores!” Jon joked.

But this perfect model also needed to be lit in perfect coherence to the on-set instruments and light sources. So additionally to the high res long-lat-HDRs that were taken on set, there were extensive survey data on each shot of all the light sources and scene geometry so that the HDRI could be mapped back in Maya onto this surveyed geometry.

The maquette of the head was photographed in LightStage with light from all possible directions (separately). A script then made it possible to color and blend these separate light-passes together based on the information of the on-set HDRIs. Why the hassle? Because the renderings were put next to this near perfect reference and the artists could check on how close they got.

The next obstacle was to choose the right approach on how the HDR sampling should be done, either Inside-Out (I-O) or Outside-In (O-I) from the HDR. The I-O approach is usually used to sample the environment for Global Illumination. You have to fire a lot of rays to cover correctly bright light sources. So you need to find hot spots and treat them as emissions. I-O works well with spheres but with other geometry you get shadow bending ⁴. The solution to this problem was to scatter the origin of the emission-positions during rendering (see the paper of Kollig & Keller, 2003).
probably guess that it didn’t simplify things that the head was moving through the scene.

The solution to all this blocking and head-movement was to reposition the HRDIs on every frame on the position of the body-double’s head. Because there was enough tracking data of the head moving through the scene the mapped HDRI in Maya was rendered in Nuke to match the position of the head which was much easier than doing it from scratch.

What comes now is really sexy: To single out light sources the direct practicals and instruments visible in the HDR were blocked or painted out in Nuke resulting in an HDR image of the ambient lighting. The missing “hero lights” were then positioned as area lights in Maya and given a HDRI texture. This was also very important for the eye-lights.

Still there had to be adjustments made for the eye sockets and eye-lights because on set the lighting was done on the body actors.

I really realized that I want to work at Digital Domain: They value Maya, Mental Ray and, most important Nuke. Adopt me!

Blogging,
originally uploaded by Phil Strahl.

The last presentation I saw before going to my car for some sleep was by Steve Preeg on the Animation and Performance of Benjamin Button. The big issue was on how to capture the performance of Brad Pit and have it applied to the digital head. And because the show was a $ 150 million Fincher/Pitt movie there was no room for error. If you’d mess it up, they would mess you up.
To get all the muscles in Brad Pitt’s face right Preeg thought about CAT scanning him but his manager just told Steve to think of something different. And so he did. Initially Digital Domain got the guys from Mobile who had developed a volumetric capturing system and captured various key poses of Brad Pitts face as basis for the blend shapes in Maya. When everything was tested and worked on they needed to capture the actual performance by Brad for the digital head.
They had him watch the clips from the movie with the body actor so he knew what was going on around the him. During his performance his face was filmed from four different positions, his cues were given brad via in-ear monitoring. In fact, Digital Domain even tried Imagemetrics but the result was too ‘dead’ to them, however it helped much in timing the animation which was all done by hand. Thus it was guaranteed to keep the intent of the performance rather than applying it with strange results. “Sometimes is just a millimeter more or less on one of the eyelids between creepy and cute”.

What I have learned today:

• That Steve from Imagemetrics probably wouldn’t pass a Turing test.
• That you can capture the facial performance of actors during motion capture by having them wear head-mounted camera-rigs with a light source both pointed at their faces.
• That on a Z-buffer approach to subsurface scattering the resolution of the buffers matters a lot (bigger = better).
• That on a Z-buffer approach to subsurface scattering you should keep the buffers separate, meaning that nothing that’s not part of the skin may cast shadows inside the skin.
• That on a Z-buffer approach to subsurface scattering won’t let you have your precious raytracing. So nobody does it anymore.
• That you best take texture photographs of skin by having polarization filters on your lights and one (90° out of phase) on your camera. Thus you block out the specular highlights and only get the diffuse light. Still you need to paint out shadows. Use 6 soft lights when you don’t have the luxury of having a Light Stage.
• A big deal in believability in CG skin are oil layer and peach fuzz. If you can’t nail it down why something doesn’t feel right then it’s usually one of those things.
• That working on 64 bit machines with 16 gigs of RAM really saved Digital Domain’s ass in producing Benjamin Button.
• That the UV-Space in Nuke (if kept in the EXRs) can save much time for last minute changes on textures.

What surprised me today:

• That you can talk passionately about human emotion without showing any.
• That Image Metrics also use Eurostile as their house font. Like me. And they’re not the only ones so I really should think of a new font then…
• That relatively few people who want to work at Pixar are interested in lighting, shading, layouting, rendering, controlling, software engineering or cinematography. They all want to become animators, character designers or, cough, directors.
• That I used working with z-buffered renders a lot in the hey-days of the late 90′s. I feel old.
• That not a single CG spotlight was used for the lighting of Benjamin Button.
• That Brad Pitt’s teeth were too white to pass as a 70-year old. For the digital head Steve Preegs teeth-color was used. That’s why he quit smoking on the show.
• That it was the first time that I read ‘LOL’ in a presentation. It is 2009 and netspeak finally conquers offline-speech.

You ask “What’s new? What’s cool?” and I tell you: A lot: I’ve been in the trenches with Nuke and fought After Effects so there’s a lot of stuff I want to show and tell what I’ve learned in the past weeks, not only about VFX.

I recorded a couple of tracks for my upcoming album (release: summer 2009). The Samson G-Track is a sweet piece of hardware, it combines a condenser microphone and an USB-soundcard. Finally I am able to record my acoustic guitar and piano work without the “help” of my 5€-headset whose microphone buzzes worse than the wasp hive in Donkey Kong Country 2 and rumbles more than my PS2′s Dualshock 2 controller that surrendered yesterday to material fatigue after nearly nine years of heavy duty service. Got me a new one today.

Between all my private creative work I am tackling 87 effect shots for our student short film “MOSKAU” and have only 4 weeks more to go. Luckily I am not alone with that truckload of effects still many shots end up on my todo-list. I am really honing my skills with Nuke right now. One year ago I only considered After Effects as the way to go and even was surprised that neither Double Negative nor Dreamworks ever came back to me after my enthusiastic applications at the fmx/08 conference.

That reminds me: If I was you (and I am 80% certain that I am not) I would keep an eye on this blog because I’ll be covering the fmx/09 in Stuttgart in detail on this very blog, complete with pictures and serious grammatical errors. So if you won’t be able to see it all in Stuttgart yourself, come back on the 5th of April 2009.

That’s all for now folks. I gotta get some sleep before the weekly meeting tomorrow. Which means I am shredding some Guitar Hero songs on extreme. Yes, you read correctly: I am publicly admitting that I am good at that game. On the upside: I got better on a real bass guitar as well.

So it all boils down to that simple phrase for me at the moment: Practice makes less imperfect.

I was plotting and scheming how I could convert a HFS+ file system into a platform friendly FAT32 but the easiest way is usually the one which takes you the longest. I copied every single file to the Mac’s hard drive, reformatted the disk to FAT32 and currently I am waiting as the files are transferred back onto the external hard drive.

Effectively this whole process ate up about one hour of my time. One hour I could have been working on the data instead of moving it around. But it wasn’t over already.

I remember requesting the files as, quote, “MXF files, straight from the P2 card” from my source but I haven’t paid attention to the fact that I won’t be supplied with dailies rather than excerpts from an edit. So I got the strangest files I’ve encountered so far: MXF.DV-QuickTimes that won’t play back on my PC – thanks to DVCPRO-HD compression. I strolled the web to get hold of the codec for QuickTime. To cut a long story short: For Windows users there aren’t any. So I spent the rest of the evening in the lab on a Mac with Final Cut Pro installed which read and played back the files no problemo re-encoding and reading some gaming magazine.

So it was recoding time – from HDVCPRO HD to QuickTime JPEG with about twice the file size that made me fondly remember the days back when I purchased my notebook — when 40 gigs meant a real big hard drive.

And now?

Now I’m reformatting the QuickTime files to TIF image sequences because Nuke doesn’t like JPEG QuickTime files. Time lost today: About 7.5 hrs — and counting.

What I have learned today:

• That sometimes it’s the cable, not the device.
• That you can access a Mac’s Disk Manager under Applications > Utilities.
• That this Mac can read NTFS disks — but not move data onto them.
• That I really can’t stand a Mac mouse’s acceleration and behavior.
• That there’s no way to read a DVCPRO HD QuickTime file on a PC.
• That a Mac has to have Final Cut Pro installed to play back a DVCPRO HD QuickTime file.
• That QuickTime exports the video files in the resolution currently displayed.
• That JPEG-QuickTimes (maximum quality) are twice as big as eqivalent DVCPRO HD QuickTime files.
• That Nuke doesn’t like QuickTimes with JPEG compression.
• That I should communicate better what files I need in what format and on what file system.

Another convention report, my apologies! I haven’t thought that I would be on so many (= two) conventions in the field of digital film in such a short span of time. But attending the Digitale Cinematographie convention this Thursday in Munich was something way out of the ordinary because a little dream never dreamed came true: Seeing my work on the grandeur of a real IMAX theater silver screen.

Till,
originally uploaded by Phil Strahl.

I have to yell out a big thanks to the head of the video department at the FH Salzburg, Till Fuhrmeister right here in the beginning. Thanks to his good connections he managed to squeeze four of my class’ video productions into the screening at the Digitale Cinematografie. And it wasn’t just a “normal” screening on a standard video beamer in some cheesy seminar room, not it was a HDCAM tape screened onto a huge screen in a former IMAX theater. Impressive!

With Zorica “Zoki” Vilotic, director and co-director of two of the screened productions, I drove fairly early to Munich and arrived perfectly in time at the convention center. Till was already there as was Sebastian Prittwitz, classmate and director of “Zugzwang” with his cinematographer of choice, Kaspar Kaven. After getting our name tags and stocking up on various giveaways Zoki and I headed for the FH’s desk which was on the floor below ground level, riding the escalator felt a little like descending into a parking garage.

Between the countless and immeasurable expensive digital cameras there was a small tables with an iBook and two seats, representing the Fachhochschule Salzburg. While Till and Sebastian were gone getting registered at the front desk, something they totally forgot about, Zoki and I took seat and represented the FH which nobody was interested in.

Zoki and me,
originally uploaded by Phil Strahl.

Nobody? Well, that’s not quite correct: One man, a little out of breath, desperately looking for something, skimmed the stands reading the various FH majors such as orthoptics, forest products & timber constructions or midwifery, then took a glimpse at the FH logo. He turned to us and asked with hope in his voice

Zoki and I both pointed at the brightly lit cubicle on the other side of the floor with plasma screens and red carpet and replied uni sono

He smiled, thanked us and off he went. The FH-logo doesn’t look that much like the RED logo, though…

It turned 10:35, ten minutes to go for our grand premiere so Till packed his Notebook and we entered the IMAX theater, getting comfortable while students from the HFF Munich where talking about their previously shown film, shot of 35mm film stock.

Things that hurt

We remembered what Till told us after he came back from mastering our short films on HDCAM: They were all too grainy, they weren’t color graded well and they wouldn’t stand against real 35mm film in a theater. I should point out that none of our productions was natively shot in real 1920 by 1280 full HD:

Visit Zugzwang

Sebastian “Basti” Prittwitz’ short film Zugzwang, which had cashed in quite a lot of awards, was shot on HDV but it really did well competing with big movies shot on film. Basti was very lucky having Kaspar Kaven as a director of photography who lifted an above-average student film to an even higher level. The trailer carried a wide range of emotions and scenes, more like the trailer to a full length feature film.

RADO commercial

We claimed that Jonny Hennebichler’s big commercial for Rado (which consisted solely of tricky chroma keying shots, multiplied to on countless layers in After Effecs) was shot in HDV. The truth is, that the crazy folks shot it in DV PAL! I don’t know how the managed to do it but it looked even better than HDV! And Andi Leitner’s music really gets close to you and grips your subconscience with a catchy melody.

Wii Monastery at YouTube.

Zoki’s Wii commercial was shot on DVCPRO HD, yes, but because of the format of half HD (1280 by 720) I cropped nearly every shot in the post production to this format, mostly because of the heavy motion stabilization that was necessary. For the presentation I had to blow up the whole thing to 1920 by 1080 and, hell, it was blurry! To compensate for the loss in detail I added some more grain to it, so that the noise was in fact the only thing that really was full HD. If asked about the terrible grain I would state that it was the artistic desicion of the director to achieve a 16mm look.
Only my matte painting was executed in full HD but in comparison it looked way too crisp. So to match the bluriness of the blown up live-action footage I had to blur the matte painting, which really hurt, until it blended together well. The additional grain I had to add hurt even more.

The Instant 36 film festival opener, titled “Motel 36″, directed by Zoki and Peter “Pepe” Pflaum, on the other hand was shot and produced in full HD ¹ and looked gorgeous on my screen during the post. For the film festival it had to be scaled down to PAL, even PAL 4:3 letterboxed which also hurt. The Digitale Cinematographie convention provided the one and only chance to show the trailer like it was meant to be presented: In a theater on a real big screen in its native resolution. I don’t have to add that I am really proud of how well it turned out and how well it conveyed the fifties film-noir look, do I?
Fun fact: Kerschy and Max (the guys I was in the Windshield team with) didn’t believe that we really shot the scenes but took them from some old movie. I think Max still isn’t convinced that we really shot it totally ourselves. Although it wasn’t meant to be one, I took that as a huge compliment for my skills in cinematography, editing and compositing. Thanks to you, all you doubters!

15 minutes of fame

Till & Su Turhan,
originally uploaded by Phil Strahl.

Presenter Su Turhan announced the next fifteen minute block featuring projects from the FH Salzburg, asked Till down, who apologized for the bad visual quality of the upcoming shorts. Then the lights went out and my heart was pounding: There it was, the instant 36 trailer many of us had worked so long on on the real big screen with real good sound, followed by Rado (which was but a little blurry but posed otherwise no evidence of its DV heritage), then Wii and rounded off well by the Zugzwang trailer. Till was mistaken: Our projects didn’t look like the crappy pixely digital video we all were terrified of to anticipate.

Very polite applause as Till and the moderator asked us down for a couple of questions, like what format we shot on, whether we even think well of traditional film and so on. Then it was over and when leaving the theater all of us wore big smiles, talked too loud and too fast to eachother and were just so very happy that we had the onetime chance of seeing our works in such a magnificent way.

Ship of interns

The following hour we roamed around the exhibition space, collected giveaways, tried out cameras no one of us will ever be able to afford in the near future and talked about how shitty 3D looks in live broadcast.

a man with glasses said who came out of the presentation space of the cubicle of dvc. It was Jürgen Firsching, managing director. We chatted a bit about stereoscopy and compositing, that Nuke 5 is finally able to handle stereo footage and where I come from.

He mentioned some names, unfamiliar to me, what they did within the company and he gave me his card along with a DVD titled “The compositor’s Introduction to Nuke”. I already knew that the tasks at dve don’t interest me much but I told him that I will think about it nevertheless. And I am really happy about the DVD because the sooner I switch from After Effects to some real compositing tool the better. And Nuke has always been on my radar.

In fact, that was the day. To round it off I bought myself four comfy cotton t-shirts at American Apparel and a venti caramel macchiato at Starbucks. Then Zoki and I went back to Salzburg at around noon.

What I have learned today:

Giveaways,
originally uploaded by Phil Strahl.

• That the FH logo looks a tiny little bit like the logo of the RED cam.
• That the best delivery format for a HDCAM mastering is providing a Targa image sequence with uncompressed audio in 48khz.
• That stereoscopy still poses a problem for live broadcast or sport — and especially for live sports casts.
• That conventions combine both the sophisticated nature of man (gaining wisdom) as well as the archaic (hunting down/collecting giveaways).
• That I am close to having a matching keychain for any of my hawaii shirts.