Reconstruction Problems

I've converted most of my test sets to SIFT keypoints so that I can work with more accurate matches. However, I'm having a lot of trouble implementing the projective reconstruction stage. It took me longer than expected to implement the algorithm for a 2-view projective reconstruction and I know I've made a mistake somewhere because the results look like gibberish. One problem is the amount of guesswork involved when using uncalibrated cameras/images -- one has to guess a camera calibration matrix (focal distance, skew, etc), and rotation/translation between cameras before the 3d coordinates can be extracted. When these are unknown, the algorithm involves a lot of variable tweaking. I prefer that my program require as little user interaction as possible.

I hope to get projective reconstruction working soon, but after that I still have to upgrade it to a euclidean reconstruction. With only a few weeks left in the quarter, I'm concerned that I may not have enough time to work on visualization, which is the main focus of my project...

For reference, here is what my "gibberish" looks like. I've plotted the results of my projective reconstruction: the red dots are the 3d coordinates as viewed from the front (same as the image), the green dots are from the top, and the blue dots are from right side:


As you can see, the front view looks nothing like the original image. The top and side views have points on the opposite sides when there shouldn't be any.

This is the first view that they were generated from (green dots indicate which ones were used by RANSAC to get the fundamental matrix):


I'm using the "canonical decomposition" of the fundamental matrix I estimated earlier -- which means the first view's camera projection is initialized to the identity and the second view's camera projection is displaced from the first one. Once the camera projections are known, the 3d coordinates of the projective structure are extracted.