Getting the Data in
The step of getting the actual data into the CG package is actually the easiest part of the process, though still prone to a variety of errors. It seems that no 2 CG or Motion Control systems have exactly the same co-ordinate system. For many, Z is forwards, but for some X is forwards; some have Y up, some Y to the left and some Y to the right. If you are importing data into a CG package, make sure that you have a system for matching the axes, or use a script or tool to get them right. In addition, make sure that it converts the units from the import file into your units. Camera Control, Inc. has a MEL script that will do this for Maya for any Flair Motion Control move, and this takes the strain out of this step. Flair will be able to export the camera move in Filmbox (.fbx) format shortly and this should assist with other packages that do have Filmbox plugins (3DS Max etc)
Scale
It is also important to have the CG world set up to match the real world. For example if you are building a doorway that is 80 inches high, make sure the units you are working in are inches and that the doorway is 80 units high. I have seen doors built in CG that are 80cm high!
Most CG packages support different types of camera. The best camera to use when you are importing data into a CG (or out of it for the matter) is a 2 node (camera and interest point) camera. The reason for this is that it eliminates the need to worry about pan and tilt directions, it handles some issues with the camera flipping orientation when when you pass over an object and if you have an interest point, it gives you additional information about the move. Please note that single node cameras in CG do not always have the same rotation order as a real camera (Pan, Tilt, then Roll) and this mismatch causes errors only when you are dealing with single node cameras.
Match the Worlds
Once the data is in the CG package, the CG camera will be doing the right move, but probably in the wrong place in your world. About the only thing that is common to both the real world and the CG world is that Up is vertical; the orientation of the other axes and the distances from the origin are often wildly different. The trick here is to find some way of matching the real world to the CG world so that the axes and offsets can be aligned. I have found that the best way to do this is to use the real camera to “survey” some reference point in the real world that will also exist or can be modelled in the CG world. For example if you are trying to make a CG character walk across a table while the camera moves around the table, use the real camera to survey the table corners.
Reference Points
With Flair, one simply lines the cross-hairs up onto 2 corners of the table and notes the distance from the camera to the corner. Flair’s target tracking software will then generate points in space that are at the corners of the table in Flair space. If these points are imported into the CG world, you will immediately see where the real and virtual objects are and what translations and rotations have to be applied to make the 2 worlds match. Incidentally, if you find that the sizes of the tables do not match, that is an immediate indicator that a scaling has gone wrong somewhere, and would be the first thing to fix. Unless you are doing very advanced moves, it is not a good idea to rotate about any other axis than the vertical one.
Reference points should be recorded with the camera as close to the move path as is reasonable and should be recorded in way that reduces any possible inaccuracies. For example choosing points that are further apart will reduce any error created by inaccurate measurements. It is very beneficial if the CG Supervisor or CG Artist is on set and helps determine how best to record the reference points. It also helps if their importance is conveyed to production who often have no idea what they are for or how much easier it makes the post if they are done properly. Surveying the track position or measuring the camera height above the ground is somewhat helpful, but good reference points can be all you need.
Sometimes a target survey is not practical and you may have to survey something based on the camera's orientation. For example if you were doing a move on someone standing, there are no good fixed points to survey. The points need to be fixed a set distance apart and need to have good horizontal separation as well as preferably being on the same horizontal plane. The best solution with a person would be to level the camera and put the it directly in front of the person a known distance from the person's nose (or between the eyes). This position can easily be matched in CG as long as the distance from the nodal point of the lens to the person is accurately noted.
With the Flair import MEL script, you can import a “reference move” so that Maya creates a series of Null locators for each position in the file, one for the camera, and one for the target. The MEL script also creates a parent Null. If you ONLY translate and rotate this Null, then the values for that Null (Trans X, Y, Z and Rotation about the vertical axis) will be the values that you need to apply to the move to get it into the right space. You can or course also rotate and translate the CG scene to match the motion control data.
Apertures and Focal Lengths
Motion Control does not necessarily need to know about the camera's aperture or the focal length of the lens, it is more involved with where the camera is and where it is looking. However these factors are vital to getting a good match visually. It is important to note the focal length of the lens and also the aperture used during shooting. With a film camera, the aperture is most often 0.980” by 0.735” which is 35mm full gate. With Video the aperture is the dimensions on the chip that are being used to gather the image. However, simply knowing the aperture dimensions does not fully give you the correct aperture to use in CG since the film could have been cropped when transferred or with HD the camera could have been using only a part of the chip depending on the format. For example, I have had people swear that the image was transferred Full Ap., yet a camera's aperture always has rounded corners. If you can't see the rounded corners on the transfer, then it was not transferred Full Ap.! The best answer is to keep good notes AND to also shoot an object or a known size a known distance from the front nodal point of the lens (NOT the film plane) CG Cameras seem to be modelled like pin hole camera, not with real lenses. (Tech Note: The correct point is the front nodal point or more accurately the front entrance pupil – however the difference between these is probably the least of your worries!)
Zoom Lenses:
Don't use these. If you have to, then also shoot the motion of the zoom lens with the camera stationary pointing at a grid, a known distance away. It is also a good idea to shoot a pass of the move with the zoom lens locked at a known focal length. Also make sure the Motion Control operator has accurately calibrated the zoom lens and if all else fails, make up a look up table of lens motion against focal length.
Once you have gotten this far, it should all work perfectly. If it does not line up well, DO NOT simply start adjusting settings to try to make it right. There is a point at which you can say “OK, this is as good as the imported data is going to get, it’s time to track the last little bits.” On MANY moves this will be minimal or negligible, it is really a matter if the lens, the shot and a multitude of different things. Before you have reached this point, only make changes or corrections to more accurately match the real world or the real conditions of the shoot, if you start “tweaking” here arbitrarily, so it looks a little better here at the start of the move, you will throw the rest of the move out, and not fix a basic error in the lineup.
Is the move wrong or is the focal length/aperture wrong?
If your move is doing the right thing, but the focal length or the aperture is wrong, the images will not match. If the move is in the right place and it is the right move, the line up will be right in the middle of the image, and will get worse the further from the centre you are. If it looks bad at the edges of the frame, then any error at all including lens distortion can cause that. But if it is right in the middle of the frame, then you can be pretty sure that the move data is right, and your lens or lens modeling is off.
Rule about visually matching moves:
There is a maxim that has to be followed when transferring data between virtual systems and real systems. That maxim is that you must work from known values and adjust for known and understood offsets between the systems before you start visually tracking. The converse of this is that once you start visually tracking, you have abandoned logically trying to work out the reasons why the move does not match and are stuck with visually tracking the move.
The main reasons it won't line up
Of course it won’t always line up as expected, and here are some of the reasons for that. (In no particular order)
Focal Length. The focal length of the real world lens and the focal length of the CG lens are often quite different. It is an issue for the CG artist to work this out, often it is a good idea to shoot a framing leader before the shoot, and also a cube/grid of known size at a known distance from the lens. One way to tell if the focal length is the major issue is that the line up should fall off more the further from the centre of the lens you are. If it lines up well in the middle, then probably the focal length is off. Zoom lenses add another whole world of misery.
The image has been adjusted, cropped, sized, repositioned etc. in telecine. It is sometimes a good idea to make a framing leader and take it all the way through to the final CG line. These will often get dropped out of the telecine, but they will tell you if the telecine artist has gotten unduly creative. This also applies to HD as the chip can be under-scanned or the footage could have been cropped or otherwise adjusted before you imported it in.
No motion control or real world system is perfect. No lens is perfectly centred, and no motion control system is perfectly rigid. There is always some droop to an arm, some slight misalignment in the lens to the camera body or some give in the floor under the rails. Due to this, you cannot expect it to line up matt-pass perfectly between the real and virtual worlds. Of course multiple passes will line up, but that is not what we are talking about. This is often the least of your worries, you should not assume the motion control data is bad until you have eliminated everything else (or you are working with some other motion control company!).
You have selected the wrong move or the move data that does not match the take data. This can cause seemingly incomprehensible problems and should be watched for.
The start of the data and the start of the move must line up. Often there will be a bloop light in the shot that indicates where the move starts or some known frame count of the move. If the bloop light is at the start, it is usually but not always the 1st frame of data – check with the motion control operator (It is in Ascii in the Flair file). If you do not see the bloop light, then possibly it has been edited out and you will need to find it on the telecine and make sure you know the frame offset between it and your move data.
Lens distortion.
If you have got this far and the input data is as good as you can get it. Time to start tracking.
Summary:
If you are importing motion control data into a CG package:
Have the responsible CG artist or similar on set during the shoot.
Take good notes on the size of the objects in the scene, particularly whatever you have selected to be your reference object(s).
Shoot a framing leader and make sure this gets through the telecine/transfer process. Shoot a pass of the reference points if possible or take stills showing where they are and where the camera is in relation to them.
Make sure you have accurate notes about which take is the approved take and also note, if you can, the file name used on the motion control software and the frame count of the bloop.
If you foresee a problem with the tracking or feel it is vital, tell the operator, tell the AD, and make sure you get the reference data you need to do your job. Since these items do not immediately show up in the finished spot, they will be the first to go when there is a time crunch.
Shoot a tracking pass with an object of know dimensions in frame if time allows.
Simon Wakley
Lead Operator Camera Control, Inc.
Author of Flair Motion Control Software
Notes:
Nodal Point: A CG lens is a point in space. A real lens is much more complicated. If you measure from a CG camera to an object, you are really measuring from the front nodal point of the lens to the object. Measuring from the film plane on a camera is NOT the same as measuring a camera in the CG world. Have the Motion Control operator set up and show you where the front nodal point of the lens is.
Zoom Lenses: Don’t use a zoom lens unless there is no way out. If you do, then shoot each move with the lens not zooming and also shoot the zoom action against a grid with the rig not moving. There will still be some “pin cushion” effect or “banana”-ing of the image as well as some backlash in the zoom mechanism.
Video Cameras: Almost all film cameras use a ground glass that has cross hairs. These are used to find and line up the reference points. Video cameras do not have these, and it is a good idea to get a crosshair generator in the line when you are doing this, otherwise, the centre of the video frame is very hard to accurately mark on set. Also note that many video cameras do not have the firm, rigid mount that film cameras have and it is possible for them not to be lined up accurately on the motion control rig, and also possible that they move. (This is not true of most of the high end Hi-Def cameras)