Key Concepts

• calibration
• perspective

Lecture Notes

Image Projection

World frame and camera frame

• Change of basis

\[\mathbf{X} = R\mathbf{X}_0 + T \in\mathbb{R}^3 \]

• or with homogeneous co-ordinates

\[\mathbf{X} = \begin{bmatrix} R & T \\ 0 & 1 \end{bmatrix}\cdot \mathbf{X}_0\]

Projection

• Recall the ideal perspective

\[\mathbf{x} = \begin{bmatrix}x\\y\end{bmatrix} = \frac fZ\begin{bmatrix}x\\y\end{bmatrix}\]

• In homogeneous co-ordinates we have

\[ Z \begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 & \\0 & 0 & 1 & 0\end{bmatrix} \cdot \begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}\]

• \(Z\) is typically unknown.
• We write \(\lambda (=Z)\) for this unknown constant.

\[ \lambda \begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix} \cdot \begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}\]

Ideal Camera Projection

\[ \lambda \begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}f & 0 & 0 \\0 & f & 0 \\0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix} \cdot \begin{bmatrix}R & T \\0 & 1\end{bmatrix} \cdot \begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}\]

\[ \lambda \textbf{x} = K_f\Pi_0\mathbf{X} = K_f\Pi_0g\mathbf{X}_0\]

• Canonical projection matrix \(\Pi_0\)

From meters to pixels

• Same units for world frame and image frame, i.e. meter
• Meaningless - images are measured in pixels

\[\begin{bmatrix} x_s\\y_s\end{bmatrix} = \begin{bmatrix} s_x & 0\\ 0 & s_y\end{bmatrix} \cdot \begin{bmatrix} x\\y\end{bmatrix} \]

• Each pixel is \(s_x\times s_y\) in metric units
• Then we translate the origin

\[x' = x_s + o_x \quad y' = y_s + o_y \]

• or in homogeneous co-ordinates

\[\begin{bmatrix} x'\\y'\end{bmatrix} = \begin{bmatrix} s_x & 0 & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} x\\y\\1\end{bmatrix} \]

• Sometimes the pixels are not rectangular

\[\begin{bmatrix} x'\\y'\\1\end{bmatrix} = \begin{bmatrix} s_x & s_\theta & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} x\\y\\1\end{bmatrix} \]

Summary

• Camera Intrinsic Parameters

\[\lambda\begin{bmatrix} x'\\y'\\1\end{bmatrix} = \begin{bmatrix} s_x & s_\theta & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} f & 0 & 0 \\ 0 & f & 0 \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} X\\Y\\Z\\1\end{bmatrix} \]

• Calibration is not a straight-forward matrix operation
  • It is not an operation on the luminance values of the pixels

• Instead

  1. The pixels are moved around in the plane, off the orignal grid
  2. Then we need to resample the image to get a new set of pixel in the original grid

Distortion

Calibration

• Camera Calibration as described above is a transformation in the ideal model
• In addition we need calibration to compensate for distortion = imperfection in the camera
• Two types
  • Radial Distortion - typical for wide angle lenses
  • Tangential Distortion - when image plane and lens are not parallel (ignored in the textbook)

Radial Distortion

• Artifact of wide angle lenses (wide field of view).
• Simplest effective model:

\[ \begin{align} x &= x_d(1 + a_1r^2 + a_2r^4) \\ y &= y_d(1 + a_1r^2 + a_2r^4) \\ r &= x_d^2+y_d^2 \end{align} \]

• \((x,y)\) are the true co-ordinate of some point
• \((x_d,y_d)\) are the co-ordinates in the distorted image

• This is easy to automate, so we can disregard distortion for analysis

• OpenCV uses a sixth order model

\[ \begin{align} x_d &= x(1 + a_1r^2 + a_2r^4 + a_3r^6) \\ y_d &= y(1 + a_1r^2 + a_2r^4 + a_3r^6) \end{align} \]

Tangential Distortion

Not discussed in the text book

• Occurs when the lens and image plane are not parallel

\[ \begin{align} x_d &= x+ [2p_1xy+p_2(r^2+2x^2)] y_d &= y+[p_1(r^2+2y^2)+2p_2xy] \end{align} \]

Notes

We are going to do the calibration in practice.

• OpenCV does not detect corners in all images and the tutorial recommends running a loop searching for corners while you move the chessboard
  • Sample Code
• You may have to tweak the delays so that you see what you are doing in real time. I had to fiddle a lot to get this right.