FMX ’10, Day Two
On the second day we all got late to the first lecture and missed “The VFX of Iron Man” and instead enjoyed the breakfast at our value-priced hotel whose every room was kept in shape for the whole place looked like a museum of 1970′s rustic dwelling. Mrs. Zheng apologized for not having boiled eggs and I downed every bit of orange juice that was left on the buffet because I almost died of thirst the night before. Mrs. Zheng didn’t like seeing me drinking eagerly directly out of the jar but left it at a sullen glance this time. Then we drove off to the Haus der Wirtschaft once again.
Post is Prep
Still a bit drowsy I planted my ass in the front row of the König-Karl Halle and knew I wouldn’t be getting up for a long time, not even for Pixar’s Career Gears (they don’t need compositors, I got the message last years). So at 11 a.m. “The Role of Visualization in the VFX Production” by Kevin Todd Haug, VFX Designer at FX Cartel and Ron Frankel, president of Proof Inc. begun.
And it began with some heavy, uncommented statements right in the beginning that made me gulp a little and sit there a bit shocked and concerned.
- Post is Prep.
- Compositing is dead.
- Avatar cheated.
The speakers started off with describing the status quo, that currently everything except the production itself happens in a purely digital environment, although the direction towards a fully digital approach is present also in the day-to-day life of the film-making process on set. Yet this reality asks for a paradigm shift towards the non-linear workflow of pre-production on set as well.
“All the steps from pre-production towards the reality of the live-action set are a question of creative communication.”
There have always been paradigm shifts within the VFX industry, such as the switch from analogue to digital in post production in the past twenty years. Now we are headed towards having the virtual reality (VR) of the feature not only available in pre-production and post production but also in the production itself.
Some prominent features already have arrived at that step. Whereas James Cameron used these previsualization techniques to see the virtual set through the camera while shooting, Tim Burton was offered a similar approach on the stage of Alice, yet “he just turned the screen away and preferred looking at the green-screen footage.”
“Doing an animation paradigm doesn’t solve the problem, doesn’t treat the actors in a respectful way. The question is how to bring everything into a virtual world. And what does that mean for us VFX people? [...] Most things belong in post, you shouldn’t do matte paintings afterward, they need to be known by the director before the principal [shoot], so it needs to happen in prep.” In fact quite a lot of assets get worked on (and finished) in preproduction, not only matte-paintings, but also models, textures, characters and so on. So why not use everything that’s already there on set?
Tod and Ron then showed a hands-on example of Conan or as they jokingly put it: “The adventures in Low-Budget-Land”. It was a shoot of a sci-fi-esque sequence set in the Prohodna Cave, an enormous cave near Karlukovo village in Bulgaria. “This cave is hundreds of miles away from your local
To make this work, position and settings of the live action camera needed to be tracked in real time. Looking through the camera means looking at your actors and a replacement of the green screen with the virtual set. This allows the DP to set light, angles & composition of the live action parts much better because s/he always has a near-final background when looking through the lens without the need of eventual re-lighting in post.
“It’s there to enable a creative dialog instead of people just guessing what to do and how to do it. It’s all about the discussion, not about the technical tools itself. So DP and CG artists can work together in both prep and on-set and eventually all boundaries collapse into one united effort from the beginning.”
Then the statements of before flashed across the big screen again, but this time with remarks that evaporated my concerns.
Post is Prep
The amount of work in post has no business in being in this environment. The whole point of being a Big Company assumes that you will bring a mass of problems from set to make them solve them — for a price. And that will change. The decisions will be made on set, hopefully diminishing a lot of problems. So post production will be much about “making it look nice” instead of “doing somebody else’s dirty laundry” as I like to express it.
Compositing is dead
Boy, that was a downer to me the first time I read it. Not so much a downer as “No, you won’t get a pony!”, it was much more like “There is no spoon.” Luckily Tod and Ron cleared it up what they inferred by that bold expression.
“Fixing a problem by ‘painting something out’ is not an option anymore. Since there’s an unstoppable shift towards stereoscopic features happening you have to make everything to work in stereo. And that makes a big difference when setting up a pipeline. Instead of everybody being a tiny cogwheel in the machinery, you will have more high-level artists working on shots from start to finish. And those shots will be cool!”
Avatar cheated
“Avatar isn’t really film-making. Rather it’s an animated movie with people who aren’t animators in funny suits. Essentially it is an animated movie with some live action in between.”
Wow.
Somebody really said it. But since there was all this pro-Avatar bias in the media and the industry I got a first glimpse of how the tremendous efforts of the making of Avatar where kind of equalized by the statements of some speakers.
“Everything on [Avatar ] was really expensive and if you do it again you get another Toy Story. And Pixar is way more efficient than Cameron. That paradigm just won’t work again.” And it is true: Everything on the set of Avatar only works in that particular stage and location. But in the day-to-day life of making movies (especially cost-effective) you need your equipment robust enough to travel. “The longer you stay in one location, the more you fuck it up [because] film crews really have a tendency to use up locations.”
I used the term “virtual set” quite a lot so far but Ron stressed that there is no such thing as a virtual set. “Since it is built somewhere it is real. In the end everything that’s being recorded is virtual. The bottom line is what’s on screen in the end and that’s what everybody’s there for.”
Personally I didn’t see anything strikingly new to this, although the quality and speed of displaying the virtual set though the video-playout improved over the years. But what was really impressive to me was employing (rather) simple camera projections onto simple geometry to make it possible to wander through concept art and matte paintings literally in one’s office.
And I even learned that there is a Society of Previsualization out there. Who knew?
The New Art of Virtual MoneyMoviemaking
Without much time in between Maurice Patel from Autodesk continued the previs-morning of that day. Maurice started of stressing that it is the tools you need to enable interactivity such as real-time motion capture, real-time processing and real-time playback. Since teams need to work closer together and need to collaborate bringing everything together is of key importance, since “communication is the most powerful in visualization”.
In today’s digital production there is a wide range of different elements that need to work together like art, mo-cap, previs, CGI, rendering and what you get from the practical production on location. Not until post you start putting everything together. And we all know by now that making changes in stereo is difficult if possible at all.
There’s a lot I didn’t note down what followed because in the end it was just an ad-laden lecture of Autodesk technologies and previs-services, probably nothing one wouldn’t find out by browsing their website.
What I understood the bottom-line was “Build the technology. Or buy it from Autodesk. Then work with different departments to implement it.”
Still: I see that there is a trend evolving in the industry to get things right from the beginning by employing various previs-techniques just to minimize the workload of “fixing it in post”.
Director and VFX legend John Bruno was talking about his role in the industry and being more or less the right hand of James Cameron in a number of films such as The Abyss, True Lies, Terminator 2: Judgment Day and last but definitely not least Avatar. In the talk he showed clips from each of the movies and explained some of the techniques they used. The face made of water from The Abyss for example was composited optically, in the original render it resembled a chrome-tentacle instead of being seemingly (and rather convincingly) made out of water. Unfortunately this was another lecture where I wasn’t allowed to take some pictures so you have to endue the rather boring title slide I photographed. My apologies. The next lecture by Disney’s Andy Hendrickson was titled a bit clunk “Blend (Art+Science) = Technology at Disney” but had some interesting aspects. Andy presented in the beginning a typical Disney concept art in a purplely-brown and uplifting tone, depicting a stone tower and trees with a brook in the foreground, a waterfall and very picturesque rock-formations behind it for the upcoming feature Tangled, formerly known as Rapunzel. You can see the original artwork by clicking here. The following hour Andy broke down the painting into several details such as the waterfall or the lighting which were created part by part to match the concept art as closely as possible. “At Disney it’s not so much about rendering something photo-real, it’s more about staying true to the artwork. And for that we need different technologies.” Now check the concept art from above with the final CG-version which can be seen right in the beginning of this teaser right here on YouTube. Pretty darn close, huh? “Our concept artist boasted that he could paint these artworks in the evening after having a six-pack in one or two hours, which would take us weeks to figure out on how to recreate them digitally.” Andy added with a smile. Disney uses a combination of xGen (whatever this is), RenderMan and IRender. They also plan to release their old classics in stereo and have developed a technology that more or less automatically produces usable depth maps, however they also employ 3d-models to project the 2d imagery on. Personally I am not so much a fan of Disney’s policies, economic decisions and certain aspects in their style, yet they decided to release some of their technology with open source, so that’s something new to the whole proprietary-focused industry. So I recommend taking a look at Disney’s Open Source site for the Ptex texture mapping system that doesn’t need UVs (That’s a bingo!), and I strongly recommend browsing through Disney’s publications on that site as well. Being a big fan of Neill Blomkamp’s District 9 and knowing that the movie’s VFX were comparably low-budget made me stay for the next presentation by Dan Kaufman’s “Inside District 9″. So there was not much money for the VFX, yet their quality and integration into the wild live-action plates is stunning and seamless. The design process went through many iterations, at first the shape and physiology of the aliens would allow a guy in a suit to double for an alien but this turned out to take out a lot of the anticipated realism because the audience would always be able to tell that, well, it was a guy in a suit, even if his face would have been fully CG. So the early maquettes1 defined the overall style. The final appearance of the aliens featured a more rigid exoskeleton even on the face, yet these overlapping scales were attached to the underlying geometry and animating facial expressions was not that big of a challenge. A lot of tests of various animals were made for the appearance of the eyes, in the end the team settled on a human pupil and iris but on a black eyeball. The main part of the characters was performed with Maya. In order to minimize cloth simulations their garments were applied very tight-fitting on the characters so that only certain bits needed to be simulated as dangling on the hero characters. Further they had stickers and make-up applied as an additional pass which all helped in diversifying the aliens while keeping them least troublesome in post. The compositing was done in Nuke and required a lot roto’n'paint as well as 3d-tracking and projections: Most of the shots in the movie were hand-held and had actors as stand-ins for the aliens to help the other actors as well as the cinematographer to know what was going on and where. Later, the stand-ins needed to be removed from the wild plates by geometry projections, sometimes requiring exhausting and complicated paint work. And there’s one slide of the presentation I recorded, in my opinion the most important one:
Use the simplest approach that will achieve the goal. Plan the entire pipeline Work closely with director/production Now that’s what I am talking about! Ben Grossmann from CafeFX jumped right into talking about the VFX on Martin Scorsese’s “Shutter Island”. I really liked the fact that many applied techniques were more traditional thanks to his way of movie making, nevertheless he was very open to new approaches in it. In the end there was a lot of forced perspective and model-work in the feature, composited either even in-camera or in Nuke. There’s an excellent interview with Matthew Gratzner and Ben on fxguide that I recommend to you if you are interested in the whole making-of stuff behind Shutter Island, because the lecture itself was nothing more than a narrated feature, although there is one technique described that wasn’t in the interview: Details and “tiles” of the sea crashing against New-England-shore-rocks were filmed from a crane on a location that “looked just right”. This location was shot from all angles, starting from looking directly down onto the shore until up to sky, producing what Ben called “salad bowls” of tiles that could be re-timed to match each other and be projected onto a semi-sphere in Nuke and be used in/for matte paintings. When asked about the work with Martin Scorsese Ben told that it was incredibly exhausting but totally worth it, because everybody knew that he was a legendary director and knew exactly what he was doing. In the end, so Ben said, “Marty got 200% of the VFX he asked for. For free.” Incredible, how being a legend helps in keeping the costs down in some way. Before getting his grip on Nuke Christian Kaestner was first outlining the situation of the current trend with stereoscopic pictures: They are box-office hits (e.g. Avatar) and, what’s probably most important to the industry, limits piracy because you can’t download a feature in stereo and enjoy it at home as you would in the theater (yet!). So, like it or not, it’s here to stay. But stereo is expensive. Not so much if you are dealing only with a live-action film totally shot in stereo, or a full CGI film in stereo, no, the spending gets out of proportions when you mix live-action and CGI. For example, match moving takes three and a half times as long as in a traditional show, compositing (especially scene salvage) gets at least twice as long and, alas, annoying. The on-set experience of Avatar was extraordinary for it was being in production for more than four years. The set itself was more like a full studio with all its custom-developed technology for and by James Cameron. But he needed the time to plan for single shot in detail, and also in stereo depth because you can’t afford mistakes at that scale. The previsualizations he approved became “the Bible” for how shots needed to be carried out, not even minor differences between previz and final in, say, distant mountain in the matte-painting were allowed (“and Jim got quite a temper!”). So quality control passes needed to be reviewed and green-lighted as well which essentially were the keyed live-action shots with un-shaded geometry but a 100% correct and tight track for each eye. It is often (if not always) the case that even matched lenses and cameras differ in their color fidelity and lens warping. So what needs to be done in the first place is to get rid of the lens distortion by a lens correction node. Then the problems in color can be addressed which is the tricky part: Some areas in the image match, some don’t and in the making of Avatar “a group of skilled people was locked into a room without windows for many weeks to match those kind of shots by hand”. At this point in the presentation Ben from The Foundry took over and showed some nodes of Ocula 2.1 to tackle exactly this kind of problems. Despite some of them not even being in beta, everything worked out just fine for the presentation. First Ben showed how to create a disparity map by connecting a DisparityGenerate to a O_Solver node; nothing new here: The result still is an image whose pixel values in the first two channels describe how the pixels from image A need to be transformed to become image B. Ben then set the analyzer to a single frame and added a keyframe. One can now generate image B from image A and check it against the actual image B for errors and mis-alignments; although moving objects (such as actors) should be left out. Unfortunately some flickering in can be observed, but there is a way to multisample the ColorMatcher by frame-blending the disparity for a couple of frames and/or by applying five or more ColorMatchers, each with slightly different block-sizes. Then merging them with each other by Plus, multiplying the result together and removing the original image. The result may appear a bit blocky, but it can be blurred and combined with the footage. The blurriness is almost invisible, yet this approach only works for footage and areas with low self-occlusions of the object. The Ocula 2.1 node, however, has now a multisampling-option for exactly that kind of merging a number of ColorMatchers together. A similar approach can be made to perform depth grading, generating new virtual cameras that lie between the recorded eye-positions. Lastly there is also a DisparityViewer which shows the disparity vectors. Ideally they have the same length, are horizontal and have an offset by 180° in their angle.
Abyss to Avatar
Disney’s Tools
You are not welcome here
Staying On Schedule And On Budget
Rolling Shutter?
Crosseyed Avatar
Most of the shots were composited in Nuke and rendered with Pixar’s RenderMan. Because so many studios were working with the same assets, which also partly existed as animatronics on set, all of them needed to be matched in appearance. The most complex shot in Avatar was hg016_0065, Jake Sully rolling out of the carrier after arriving on Pandora with a bypassing mech: 250,000 files needed to be rendered and were comped in a Nuke script with over 3,500 nodes. “Yet it rendered on our farm in about ten minutes. Nuke is incredibly performant!”Let’s get physical!
So if you can build a better image out of it, it works. The node even offers two options, “Normal” intended for motion and “Severe” which checks the edges, and I think it was in the O_newView or TuneDisparity node (provided the latter one even exists). The image generated from the disparity can be written to an EXR and then color matched with the “real” image from the other eye.
Comments
No comments yet! Please feel free to add one!