Revision 6811326148d6ba8f29da77309f6e9113e0c092ed (click the page title to view the current version)

Real World Reconstruction

Changes from 6811326148d6ba8f29da77309f6e9113e0c092ed to 97009778412a583b0d938693375f0e9a1ed6f442

---
title: Real World Reconstruction
---

**Briefing** [Real World Lecture]()

**Additional Reading**
[Cookbook]
Chapter 9 in  *OpenCV 3 Computer Vision with Python Cookbook* by
Alexey Spizhevoy (author).
Search for it in [Oria](https://oria.no/).  There is an e-book available.

# Exercises

To goal for the last exercise is to test a more complete 
system for 3D reconstruction, using OpenCV.  
The Cookbook by Alexey Spizhevoy is a useful starting point,
using selected recipes from Chapter 9.

You can choose if you want to record you own data,
using external webcams, or use the dataset from the cookbook.
If you want to use external cameras, you may have to collaborate
and share, but then group work is a good idea in any event.

## Using the cookbook

+ Many recipes load image data, which are available on
  [github](https://github.com/PacktPublishing/OpenCV-3-Computer-Vision-with-Python-Cookbook)
+ Many recipes load data saved in previous recipes
    + the save instruction is not shown in the book 
    + included in the complete code on
      [github](https://github.com/PacktPublishing/OpenCV-3-Computer-Vision-with-Python-Cookbook)
+ When you use the recipe, you should always ask yourself,
  *what does the code do?*
+ If there is code you do not understand, you should ask.

## Stereo Calibration.

Recipe p 240ff in the cookbook: *Stereo rig calibration*

Many features of this recipe should be known from our 
previous [Camera Calibration]() exercise.

1.  Test the recipe with your choice of data.
2.  What do the various output mean?
3.  Match the calculations in the recipe to the
    textbook theory.

## The Fundamental Matrix.

Recipe p 257ff in the cookbook: *Epipolar Geometry*

Note that this recipe does not solve the general problem
of recovering the fundamental and/or essential matrices.
It actually depends on the calibration matrix 
from *Stereo rig calibration* (above),
and it chessboard points, making the problem planar.

1.  Test the recipe with your choice of data.
2.  We already had the fundamental and essential matrices
    from *Stereo rig calibration* (above).  Do the new
    calculations match the previous ones?

## Recovering Rotation and Translation

Recipe p 259ff in the cookbook: *Essential matrix decomposition*

1.  Test the recipe with your choice of data.
2.  Find the axis and angle of rotation for each of the two candidate
    rotation matrices.  You can use the logarithm as described
    in Theorem 2.8 in the text book.
3.  Which of the two rotation matrices is correct?
    Try to answer this by visual inspection of the images.

## Triangulation

Recipe p 250f in the cookbook: *Restoring a 3D point*

Note that the recipe uses synthetic data.

1.  Test the recipe with your choice of data.
2.  How is the relative pose of the cameras defined in
    the recipe?

## Triangulation of Real Data

There seems not be be a recipe for triangulation in real data,
but we can try to make one based on all the recipes tested
so far.

### Step 1. Dataset

1.  Make a stereo camera, i.e. a fixed positioning of two cameras.
    Make sure to keep it fixed throughout data collection.
2.  Use a chessboard to make a set of images for calibration.
2.  Find some objects with a lot of corners and take images of 
    each one with your stereo setup. 

It is a good idea to take more images than you think you need, 
as it may be difficult to keep the camera setup fixed as you
fiddle with the programming.

### Step 2. Relative Pose

Obtain the relative pose $(R,T)$ of the two cameras by
repeating the recipes up until the recovery of rotation and
translation.

### Step 3. Corner Detection

Use the Harris Corner Detector to find corners in the the
object images.  (You can do it in the chessboard images, but
that is less interesting.)

If you have time, you can use SIFT to match image pairs. 
If you don't, please match them manually.

### Step 4. 3D Reconstruction

Here, we use the `cv2.triangulatePoints` as in the previous
triangulation recipe.  The challenge is to define the 
projection matrices, `P1` and `P2`, for our specific
stereo camera.

+ Recall that the projection is typically given as $\Pi=K\Pi_0g$, where
    + $K$ is the intrinsic camera matrix
    + $\Pi_0$ is the basic projection, constructed with the `np.eye` function
    + $g=[R,T]$ is the relative pose of the cameras.
+ In the *Stereo Calibration*, you have found $K$ for each camera, i.e. `Kl` and `Kr`
+ If we select the first camera frame as world frame,
    + the first camera has $g={I,0]$
    + the first camera has $g=[R,T]$ given by the relative pose of the cameras,
      as calculated in the previous recipe

Constructing the two $\Pi$ matrices, you can recover the 3D points
as in the previous Triangulation recipe.

### Step 5. Visualisation 

Take all the 3D points from the previous step and plot
them using matplotlib.  Does it look correct?

Note that we have not had time to use edge information to
reconstruct the 3D scene, and this makes the reconstruction
rather rudimentary.  It is a good start, though.

### Step 6. Reflection and Summary

Review this last exercise, and discuss,

1.  What assumption have been necessary in the work?
2.  What data do you require to do 3D reconstruction in this way?
3.  To what extent is the recipe relevant to real life?
    - i.e. what kind of problems can and can't you solve this way?