Tim Sassoon has worked on a number of projects that were shot as 2D but later converted to stereo. Here he describes the largely manual process involved.

Article first published: January/February 2008

At SFE Films Inc, we do visual effects for films, including a lot of large format and 3D films. We’re currently engaged on a very interesting project, which is the 3D conversion of a film that was originally shot in 2D, but planned from the get-go to be a 3D film. It’s called Mummies: Secrets of the Pharaohs, and it was shot by Reed Smoot ASC. We just started working on it and we have several months’ work ahead of us.

Our approach to 3D conversion of a project shot in 2D is to create two synthetic eyes. You can’t just say, “all right, we can save money by only doing one eye”, because both eyes have to match exactly. They need to go through the same path, to the point where you need to have both eyes output from the same film recorder on the same day, on the same roll of film. So thinking you could save money by reusing the old left eye is pretty much a losing game.

Applications for 3D conversion include large format cinema, location-based entertainment and feature animation. Obviously there have been quite a few animated films – Open Season, Chicken Little, Monster House – that have been converted from 2D to 3D, and that’s fairly easy to do. There’s also the opportunity for special edition feature re-releases (King Kong, Star Wars, etc, have been mentioned), but we haven’t really seen many of those. That’s what everybody thought the market would be. Also, there are co-release feature films, if the 3D can be delivered on the same day and date as the 2D, which is something we’re looking at.

Right now the market is large format cinema for live action and digital theaters for animation. We haven’t really yet seen any 3D live action films going into general release, and we haven’t really seen any really successful 3D animated films in large format either. We expect that to change with the U2 and Hannah 3D films coming out in January 2008.

The conversion process

To convert 2D to 3D, the first thing we do is look at a show that is being considered for 3D conversion and ask, “is the show a good candidate for 3D conversion?” Sometimes the answer is yes, sometimes no; sometimes it would just make boring 3D. We’re of the opinion that, in many cases, it might not be a good idea to have the same film for 2D and 3D. It may require significantly different cuts in many ways – for instance, we might want to have some surprise 3D moments, where things come out of the screen; moments that don’t really make much sense in 2D, but in 3D can really make you jump.

We then go through each shot and mark it for complexity. What we also do is have the post supervisor or director mark shots for priority. Then we can compare complexity, difficulty and priority, and if something is very high complexity and very low priority, well that’s a target for a cut or replacement or a rethink.

We pull a single frame from each shot in the show and, along with the stereo supervisor, create a 3D previs of the entire show. Every 3D movie really should have a stereo supervisor. Very often that person is also the DP, and I can mention a few noted DPs – Sean Phillips, Peter Anderson, Reed Smoot – who are very comfortable both supervising 3D and being the DP. Some of those DPs will also follow the show through post and supervise the stereo through post too.

For objects that are close to the camera, such as the sequence above from Roar: Lions of the Kalahari, we have to do full 3D match moves. We then do depth cuts. The depth cuts set the limit for 3D match moving and detailed rotoscoping, and that becomes the working pattern for the show, which can be reviewed by the director, producer, whoever. It’s just single frames, essentially in QuickTime movie form.

Lions of the Kalahari/Lions 3D

Lions 3D was originally Lions of the Kalahari, photographed (and directed) by Tim Liversedge, who was a game warden in Botswana for many years and who does a lot of work for National Geographic. We finished the show as a 2D film, but it was essentially murdered in the cinema by the 3D film Wild Safari. The producers took great umbrage to this because Wild Safari was just hanging a stereo 35mm rig off a Libra head on the back of a Land Rover and driving around a game park in South Africa looking for stuff. They didn’t actually find much, but the theaters were booking it because it was 3D.

So National Geographic rounded up some extra money and had us convert the film from 2D to 3D. We didn’t change the cut at all, and it’s the same soundtrack as the previous film, but it’s been far more successful as a 3D film than it ever was as a 2D film. It’s actually a better looking and a much more interesting film, although it was not in any way planned to be a 3D film. In fact, there are many things within it that were against stereo realization.

The above scene is what we call ‘Roto-Hell’ – although there were even more complex scenes. Top left is the mono original and next to that is the rotoscoped version. We’re false coloring the animals, so we can see which body parts belong to which. Rotoscoping is an essential process in 3D conversion. The primary task is to separate all of the objects in the scene so we can assign them each their own depth. They all need to be cut out, so we need masses of people. It’s a thankless task, and we worked for months and months on this. People often say, “surely there’s a software package out there than can automate this process”. But it stretches one to think how this could possibly be done by an automated process – it’s hard enough for the human eye to work out what’s going on.

From the rotoscoped scene we build a depth matte. In this scene we’ve also used 3D Matchmoving – building moderate complexity CGI objects which match the movements of the real animals. We do this because a lot of the animals are close together, as they’re shot on a long telephoto lens, so we want to avoid them looking like cardbard cut-outs. Using this technique, we can build a lot more depth into the objects.

This is a very rough depth cut, but it gives you an idea of where things are in space. We also produce grayscale maps along with the metadata, which are basically just numbers that represent the creative decisions. They’re very much like sewing patterns: the artist reads the metadata, looks at the depth and goes, “OK, I know what needs to be done here”.

We make a conscious effort to do everything as painlessly as possible, including creating clean plate backgrounds whenever we can – in other words, backgrounds without any of the foreground figures. We then rotoscope our foreground and middle ground objects and generate 3D models.

Once we’ve finished compositing the back plate with the 3D offsets, as well as the CG rendering, client approval and any changes, there’s the conform and color correction, which is more or less the same as for 2D, except you have to take into account projection methods. You can end up with a lot of different sets of deliverables for, say, IMAX, Real-D, Duality, active glasses, and now even for Samsung TVs. And the stereo often needs to be set up subtly differently for each one.

Tim Sassoon Tim Sassoon began his career as a cinematographer, photographing a couple of indie 16mm films before getting into motion control. After Terminator 2 he made the transition to digital. He worked for VFX shop Dreamquest for some years in the 90s, before forming his own post company, SFE Films Inc, specializing in large format and stereoscopic work.