• calibration
• perspective

# Lecture Notes

## Image Projection

### World frame and camera frame

• Change of basis

$\mathbf{X} = R\mathbf{X}_0 + T \in\mathbb{R}^3$

• or with homogeneous co-ordinates

$\mathbf{X} = \begin{bmatrix} R & T \\ 0 & 1 \end{bmatrix}\cdot \mathbf{X}_0$

### Projection

• Recall the ideal perspective

$\mathbf{x} = \begin{bmatrix}x\\y\end{bmatrix} = \frac fZ\begin{bmatrix}x\\y\end{bmatrix}$

• In homogeneous co-ordinates we have

$Z \begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 & \\0 & 0 & 1 & 0\end{bmatrix} \cdot \begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}$

• $$Z$$ is typically unknown.
• We write $$\lambda (=Z)$$ for this unknown constant.

$\lambda \begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix} \cdot \begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}$

### Ideal Camera Projection

$\lambda \begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}f & 0 & 0 \\0 & f & 0 \\0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix} \cdot \begin{bmatrix}R & T \\0 & 1\end{bmatrix} \cdot \begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}$

$\lambda \textbf{x} = K_f\Pi_0\mathbf{X} = K_f\Pi_0g\mathbf{X}_0$

• Canonical projection matrix $$\Pi_0$$

### From meters to pixels

• Same units for world frame and image frame, i.e. meter
• Meaningless - images are measured in pixels

$\begin{bmatrix} x_s\\y_s\end{bmatrix} = \begin{bmatrix} s_x & 0\\ 0 & s_y\end{bmatrix} \cdot \begin{bmatrix} x\\y\end{bmatrix}$

• Each pixel is $$s_x\times s_y$$ in metric units
• Then we translate the origin

$x' = x_s + o_x \quad y' = y_s + o_y$

• or in homogeneous co-ordinates

$\begin{bmatrix} x'\\y'\end{bmatrix} = \begin{bmatrix} s_x & 0 & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} x\\y\\1\end{bmatrix}$

• Sometimes the pixels are not rectangular

$\begin{bmatrix} x'\\y'\\1\end{bmatrix} = \begin{bmatrix} s_x & s_\theta & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} x\\y\\1\end{bmatrix}$

## Summary

• Camera Intrinsic Parameters

$\lambda\begin{bmatrix} x'\\y'\\1\end{bmatrix} = \begin{bmatrix} s_x & s_\theta & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} f & 0 & 0 \\ 0 & f & 0 \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} X\\Y\\Z\\1\end{bmatrix}$

• Calibration is not a straight-forward matrix operation
• It is not an operation on the luminance values of the pixels

1. The pixels are moved around in the plane, off the orignal grid
2. Then we need to resample the image to get a new set of pixel in the original grid

## Distortion

### Calibration

• Camera Calibration as described above is a transformation in the ideal model
• In addition we need calibration to compensate for distortion = imperfection in the camera
• Two types
• Radial Distortion - typical for wide angle lenses
• Tangential Distortion - when image plane and lens are not parallel (ignored in the textbook)

• Artifact of wide angle lenses (wide field of view).
• Simplest effective model:

\begin{align} x &= x_d(1 + a_1r^2 + a_2r^4) \\ y &= y_d(1 + a_1r^2 + a_2r^4) \\ r &= x_d^2+y_d^2 \end{align}

• $$(x,y)$$ are the true co-ordinate of some point
• $$(x_d,y_d)$$ are the co-ordinates in the distorted image
• This is easy to automate, so we can disregard distortion for analysis

• OpenCV uses a sixth order model

\begin{align} x_d &= x(1 + a_1r^2 + a_2r^4 + a_3r^6) \\ y_d &= y(1 + a_1r^2 + a_2r^4 + a_3r^6) \end{align}

## Tangential Distortion

Not discussed in the text book

• Occurs when the lens and image plane are not parallel

\begin{align} x_d &= x+ [2p_1xy+p_2(r^2+2x^2)] y_d &= y+[p_1(r^2+2y^2)+2p_2xy] \end{align}

# Notes

We are going to do the calibration in practice.

• OpenCV does not detect corners in all images and the tutorial recommends running a loop searching for corners while you move the chessboard
• You may have to tweak the delays so that you see what you are doing in real time. I had to fiddle a lot to get this right.