G1-Final Report
The Effect: Free Throw Tracking
To generate the free throw stromotion and tracking effect we first performed background subtraction to segment out the ball and player from the background. We decided to do this in Matlab since the image toolbox provides lots of existing functionality for manipulating images.
We started with some existing Matlab code for background subtraction available at: http://www.inf.ed.ac.uk/teaching/courses/av/MATLAB/TASK6/. This implementation works well with a solid black background but performed poorly on our source footage due to noise and cloud movement. We extended this implementation to build a background model from several seconds of background footage and were able to successfully extract the ball and player after applying various erosion and filtering operations.
To generate the stromotion effect we first generate a binary mask for each frame allowing us to extract pixels belonging to the ball. This binary image is produced from the background subtraction image by performing blob detection and keeping only the blob with the lowest centroid. For our camera orientation this effectively eliminates any blobs belonging to the player (in a separate step below these blobs are used to track the players position in a pickup game).
The resulting mask is produced on a frame by frame basis and allows us to build a composite image at any frame in which a variable number of balls from previous frames are overlaid with feathering (to remove the halo surrounding the ball) and decaying alpha blending (to produce the fading effect).
We also constructed lines tracking the ball's flight from frame to frame. From the blob detection for each frame we track the centroid of each blob and output a list of image coordinates. Originally, we used Matlab's built-in line drawing functionality to draw a track on the image.
To improve the visual quality of the lines and allow for greater control of their appearance we decided to use Maya to draw the lines on the image. To accomplish this we created an orthographic camera in Maya with the correct intrinsic camera properties (these were estimated by hand in this case but could also be found by calibrating the camera). By positioning the camera so that the world origin is at the upper-left corner of the image plane we were able to use the image coordinates directly from Matlab without worrying about perspective effects.
To create the lines in Maya we wrote a Python script to take the image points and frame numbers from Matlab and generate a MEL script, which creates and keyframes the lines in Maya. These frames are then rendered on top of the stromotion effect to give us the final product.
In retrospect it may have been an improvement to reorder the layers so that the lines would be beneath the ball trails but time did not allow for this extension. Our code is available here: freethrow_tracking_code.zip.
The Effect: Player Tracking
The code: imgtest.pde
The player tracking effect involved placing a nascar-style arrow on top of footage of a one-on-one basketball game.
Like in the free throw shooting segment, the first step in the pipeline involved doing background subtraction on the video of the one-on-one game segments. Using the same code as in the free throw tracking for this portion, the following image could be generated:
This was done frame by frame to get a background subtracted video.
The next step was to obtain bounding boxes for the player blobs. We obtained this using processing and a library called Blob Detection (found here: http://www.v3ga.net/processing/BlobDetection/). However, the coordinates for the center of the player varied greatly. Therefore, an algorithm to compute the weighted center of a player blob was used. This algorithm would add a column into the total if and only if the column of the player blob contained more than a chosen threshold of non black pixels. Then, averaging only these columns, the weighted axis was found. Notice the green axis here is not in the true center:
The problem of occlusion was still not solved. To deal with this, further analysis was done on the single blob that remained when the two players were close to each other. Using the same weighted center algorithm from above, the center of the blob containing both players was found:
Then, dividing this blob in half, the team tried many different algorithms to try to decipher which player was which. First, an algorithm that compared each player to their average blob color over the course of the entire video using the 3D distance formula as a metric was attempted. This produced extremely wild results. Another algorithm was attempted that compared the blob to a predetermined color (a yellowish color for Daniel and a near-black for Rob). Again, this did not produce good results. Finally, an algorithm that simply checked which of the two weighted halves of the single blob had the most “near black” pixels (i.e. pixels that were black but were not part of the background subtracted area) was assumed to be Rob, and the other area was assumed to be Daniel. Then, the weighted center algorithm was done on each of these blobs to assign coordinates to the player. This produced the following results:
Finally, to clean up the effect, a text file (example here: sequence1.txt) containing lines with a frame number, player label, and (x, y) pixel coordinate pairs was generated from the code. Finally, these were fed in to a python script (found here: tracking.py) similar to the one for the free throw effect that would generate a mel script (example here: animatePCards.mel) that would produce a much nicer looking arrow. Player names created in photoshop were placed on top of the resulting video in after effects to create the final effect.
What could have gone better:
- The tracking was jumpy at times due to the non-uniformity of the blobs. This was due to the background subtraction algorithm not being perfected. Mostly, it was because the footage we shot outside was not quite uniform due to sunlight differences and cloud movement. However, more ways to smooth the algorithm could have made the algorithm better.
- Another camera could have been used to find the true position of the players on the court. Better yet, an overhead view would have made for more accurate tracking since jumping was our algorithm's biggest enemy.
What went well:
- The hack used for occlusion detection actually worked quite well.
- The arrows in maya were a big improvement.
The Effect: Shot Chart
For the shot chart we made a virtual court in Maya. Crosses were also made and placed where the player shot.
A plain black and white image of a court that also had court dimensions was used for the lines of the court. The dimensions were used to properly scale a plane in Maya. Then a pattern was made using Photoshop's pattern maker for the wood of the court.
This was used as a texture to apply to the plane overlaid on the real court in Maya. After adjusting the texture control points the texture lined up correctly with the real court.
The plane was turned invisible while the players were on top. Then an overhead view with both crosses came in later to show another angle of the players' shots.
cross.tif
2x008.tif
The above images were used in sequence and were then composited over the video of the game.
We were going to put the players on top of the virtual court, but unfortunately after doing one frame we realized it required too much hand holding and wasn't worth the trouble.
Contributions List
Daniel
- Compositing in premiere
- Primary document writer
- Camera capturing
- Coding of the tracking effect
- Narration for the making of
Aaron
- Primary coder of free throw effect and background subtraction
- Coded the script to generate MEL commands
- Making stuff look pretty in maya
Rob
- Player Card effect
- Shot chart effect
- Compositing
Fat Joe and Lil' Wayne
Irfan Essa and Jerry Choi
Link to this Page
- DVFX 2007 Groups last edited on 2 May 2007 at 12:51 am by coppola-win.cc.gatech.edu