Download:

The story of our video is simple: life inspires art, art inspires life. It is a palindrome that holds true in many areas of life. Art can be found almost anywhere one looks. People create art based on their experiences, thoughts, imaginations, and fantasies. These same people also live to create art. Those who do not create art often find inspiration in it. Whether it is in a gallery, on television, or on a computer, art does inspire people to live, but without people, art would have no life.

This story, while a thoughtful one, was not our first... not even close. Our story has undergone several revisions, to use the word loosely. The following descriptions are summaries of some of the ideas we came up with while brainstorming for what we would ultimately become "Life in Paints":

• An Artist's Mind: An artist's girlfriend cannot understand what he sees in his work. She sees ordinary, flat paintings while he sees rich 3D worlds. This storyline was our first and most elaborated. The storyboards found on our earlier update pages reflect this one. It was ultimately scrapped in favor of a simpler one.


• Crime Scene: A forensic scientist tries to solve a crime. Shots recursively fly into pictures of the crime scene and photographs and paintings within each other in order to find the key piece of evidence.


• Newton: Isaac Newton is sitting underneath his tree, looking at his canvas, when an apple falls, hits his head, and enters the scene. He then comes to discover something other than gravity (hopefully something humorous).


• Fantasy dream: An office worker sits in a cubicle fantasizing of being on a vacation. The scene moves through postcards and paintings within each other seamlessly ultimately ending at a dream desintation, only to get yanked back to reality, playing the whole sequence rapidly in reverse.


• Domestic disturbance: The shot continuously moves through the same scene telling the story of a child coming home from school facing his increasingly violent parents. Each time the scene is revisited, the story progresses. The ending would contain some kind of relief for the kid.

Clearly our story has changed greatly over time. We decided to film the Newton scene because of its simplicity. We shot it with fairly generic acting and motions in order to give us freedom to change the flow and story later... which is exactly what we did. The footage we ended up using for "Life in Paints" was originally intended to follow the "Newton" storyline.

The above diagram shows the process by which our effects shots were created. It begins by manipulating the chosen painting in Photoshop to create separate background and forgeround images. The background layer is loaded into GTTiP and coordinates creating a 3D "tourable" painting are output. VIPeR is used to track the 2D movement of the checkerboard pattern from the video frames. This tracking information is used in Matlab to extract both camera intrinsic data and 3D camera movements. The data from GTTiP and Matlab is then given to MelSG to create a Maya MEL script that creates a Maya scene containing the 3D painting and camera movements. The Maya scene is then tweaked and foreground objects are added. The scene is then rendered. The original footage and rendered scenes are composited together in Premiere. Finally, an extra effect, birds flying, is animated and added using After Effects.

After a painting was chosen, foreground and background elements were seperated using Photoshop and saved to separate TIFF files. Each element was scaled to correct dimensions. The holes left behind the foreground elements were filled in by hand.

Download:
PSD of painting with layers (7.6 mb, Photoshop PSD)

The background image was loaded into the GTTiP program, canvas size set, and backplane coordinates placed on the image. The data is saved out in a format usable by our MelSG program.

Download: GTTiP04182003.zip (1.4mb, Mac OS X, source and binary)

A modified VIPeR Toolkit is used to track the motion of the calibration checkerboard in the video. It automatically detects the inner checkerboard pattern using OpenCV function calls and outputs the coordinates of the four outer corners of this inner checkerboard for use in Matlab.

Download:
MotionTracking.zip (1.6 mb, Windows VIPeR Toolkit, source)

The Camera Calibration Toolbox in Matlab is used to obtain camera intrinsics first using a set of pre-defined images featuring the checkerboard at extreme and varying angles. The Toolbox was modified to read in the file output from the modified VIPeR Toolkit identifying the four corners of of the inner checkerboard in each of the frames of the source video. Camera extrinsics (translation and rotation) are found automatically by using this output file.

Download (same as above):
MotionTracking.zip (1.6 mb, Matlab, source)

MELSG stands for Mel Script Generator. Its an intermediate program between the data and Maya. The program takes in motion tracking 3D data, TIP cube and image data, as well as some settings. At a click of a button the program generates the Mel script for the entire shot ready to be read in by Maya.

Download:
MELSGv4302003.zip (1.4 Mb, Mac OS X 10.2, source and binary)
Sample MEL Script (42 kb, Maya, script)

The script from MelSG was read into Maya and the 3D scene is built, complete with 3D painting and motion-tracked camera movements. Although most of the scene is automatically built, foreground elements must be added by hand.

Download:
Sample Maya scene (100 kb, Maya)

Premiere was used to edit the footage together without the effects. Once the effects finished rendering in Maya, they were imported into Premiere and composited onto the original footage. Titles and audio were also added there.

The birds were drawn frame-by-frame in Photoshop and animated in After Effects. The animation consists of not only translating and rotating the birds movement relative to the shot, but timing the flapping of the birds' wings as well.