G1: Final Report
The Effect
The effect is a simple face deformation in 2D, applied to a movie sequence. The effect can be likened to a Kai's Power Goo (or the GIMP's Iwarp filter) for moving video. Once the initial steps are taken to process the video, the user can specify interactively any facial transformation, keyframe it (all using Maya), and output the effect.
Process
First, we film the raw footage. We used an 11 second sequence with our actress moving minimally. We tested a few takes, and found that minimal movement works best for our effect. It also helps to face forward and have the actor/actress's face framed close-up. We take this raw footage and output it to series of frames, using the Canopus DV Raptor hardware/software and Adobe Premiere's export to bmp sequence features.
We take the first frame into Adobe Photoshop, and use a mask we generated based off of Dr. Stephen Marquadt's Golden Ratio Facial Mask. For more information on this, check his website, at: The Marquadt Beauty Analysis Main Site.
View Full Size
We then deformed this mask to fit our actress's face. We line it up to certain features. This mask can be transformed appropriately to fit any face.
We output this first frame with the mask overlayed (and use it as frame 1 in the movie sequence). We take this frame, and the sequence of frames to the picker program.
Picker Program
We wrote the Picker program in OpenCV, using the Hawk scripting tool to run our code on the machines at both home and in the State's lab at the College of Computing.
The program uses the Lucas-Kanade algorithm to calculate Optical flow. It essentially follows the "flow" of pixels, allowing it to "track" the moving actress's face as it moves.
The program, upon execution, loads up the first frame as a picture, and allows the user to select points to follow. Right now, we have it hardcoded to tell the user to select certain points first, so that it can calculate accurately the rotation and scaling for the mask (used later). After picking the first 6 points, the user can then select the rest of the points (the vertices of the polygon mesh on the face) in any order. This may take a while to do, but it is simple. Once done, the user can then hit R to run the program.
View Full Size
It then runs through the sequence of frames and calculates the transformation by using the average movement of all the points. It uses the first 6 points to calculate scaling and rotation. Because we know where these points are in relation to them, we set can find the rotation and scaling by relating their movement to each other.
See the code and comments for more information on how it works.
Here are some of the hardcoded values you can change:
//boolean values
#define MAKEAVI 0 //if true, overlays tracked points into an avi
#define TRANSLATE 1 //if true, creates script to translate
#define SCALE 0 //if true, creates script to scale
#define ROTATE 1 //if true, creates script to rotate
#define USE_IMAGE_AS_FIRST_FRAME 1 //if true, uses an image as the first frame
//coordinate scale of video/image in 3D coordinate system
#define XSCALE 30
#define YSCALE 20
//settings for keyframes
#define FREQUENCY 1 //how often a key frame is generated, the 2nd and last frames are always generated
//number of points in mesh
#define POINTS 69
//names of objects in maya
char* mesh = (char*)"polySurface1";
char* meshuv = (char*)"polySurface1.map[0:68]";
The program will then generate an avi (if specified) and a giant MEL script that will move a polygonal object named (polyshape1, easily changeable as a variable in the picker program) in Maya to match movement. It also moves by a scalar amount (correctly) the UV mask that lines up with the polygon mask. It also keyframes on every frame (this is changeable, how often you would like to keyframe as a variable) in both the polygon mask and UV mask.
Side Note on how the Effect Works
The matching of the UV textures with the polygon mesh is how the effect is achieved, we can move the polygon as long as the texture behind it is the exact same, or pulling textures from the exact same location. If we deform the polygon, and keep the UV texture in the same location, the polygon instead stretches the texture from the same area instead of repeating or missing data.
Next Step, Maya
In Maya, we first set up the scene. We need the image plane to be the movie sequence in the front camera view. For information on how to do this (as we figured it out and posted it to the co-web), check here.
We then used the create polygon tool to simply create a polygon over the background image plane (to match exactly the polygon applied directly on the face). This is easy with the template as a background, and after a few tries, we got really good at creating this polygon mask. We used the split vertices tools and append vertices tools as well to save time.
After this polygon is created, we group it as an object, and apply the movie as a texture map. We create a new Blinn texture material and set the texture as our sequence of images. Now comes a tricky part. The texture is not aligned correctly with the polygon. We must go in to the UV Texture Editor in Maya and manually match the polygon mask's UV vertices with the polygon mask's original vertices. This can be done easily by a few rotations (and we were working on a way to have the maya script do the lining up of the vertices, but this part was easy to do by hand at first).
View Full Size
So then once the UV texture is lined up to the polygon, we then added lighting to the scene to match the lighting of the footage. After a few tries (very few actually), we succeeded in imitating quite well the lighting by adding and arranging an ambient light some distance from the mask with a fractional intensity.
View Full Size
We then run the MEL script and wait, as it first changes frames, selects the polgyon mask and moves, scales, and rotates the mask to match the movement. It calculates a UV-space translation, scaling, and movement and applies this to the UV polygon mask (called a UV mapping). It then keyframes both the entire sequence and the UV Texture editor mapping.
After this is complete, the creative part comes in. We can simply select any portion of the face (by using the select vertices tool) and apply scaling, translation, and rotation to the features, key frame them, and maya will interpolate the movements among frames. Since the MEL script has simply moved the entire object and the UV mappings, any changes we do to certain vertices will not affect the polygon-to-UV-mapping. Hence, the effect is achieved.
We let maya render it out in 720x480 with the customized settings for DV and output the sequence as a movie. Wallah.
Tweaking
We were not happy with the way the texture mapped on the face in some areas, as it became jagged and left small artifacts, especially when pulling texture data from areas off the face. So, we used photoshop's Blur tool to retouch frames to blend the texture in at the edges.
We also tweaked the polygon mask first in some areas (very few frames) to make sure it didn't pull textures from areas outside of the face, but when this could not be avoided, we used the photoshop tool described above to fix these minor areas of detail.
Areas of Improvement
We think that using more points in the calculation could improve the effect.
There was some error in the calculations that built up, and as the MEL script progressed and moved, this error propagated since each calculation built off the last.
We think that future versions could even operate in 3D, instead of performing 2D scaling, rotation and translation on a 2D mask, we could apply 3D scaling, rotation, and translation on a 3D mask.
Necessary Files for Effect Generation
picker2.c // Version 2: Uses average of 6 points
picker.c // Version 1: Uses all points, CPU-intensive!
script.mel
Frame.bmp
script.mel
Link to this Page