Revision 97dab15eecbb2983a0225555bf8d4a875e1fca32 (click the page title to view the current version)
Changes from 97dab15eecbb2983a0225555bf8d4a875e1fca32 to current
---
title: Camera Mathematics
categories: lecture
---
# Key Concepts
- calibration
- perspective
# Lecture Notes
## Image Projection
### World frame and camera frame
+ Change of basis
$$\mathbf{X} = R\mathbf{X}_0 + T \in\mathbb{R}^3 $$
+ or with homogeneous co-ordinates
$$\mathbf{X} = \begin{bmatrix} R & T \\ 0 & 1 \end{bmatrix}\cdot \mathbf{X}_0$$
### Projection
+ Recall the ideal perspective
$$\mathbf{x} = \begin{bmatrix}x\\y\end{bmatrix}
= \frac fZ\begin{bmatrix}x\\y\end{bmatrix}$$
+ In homogeneous co-ordinates we have
$$ Z \begin{bmatrix}x\\y\\1\end{bmatrix} =
\begin{bmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 & \\0 & 0 & 1 & 0\end{bmatrix} \cdot
\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}$$
+ $Z$ is typically unknown.
+ We write $\lambda (=Z)$ for this unknown constant.
$$ \lambda \begin{bmatrix}x\\y\\1\end{bmatrix} =
\begin{bmatrix}f & 0 & 0 & 0 \\0 & f & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix} \cdot
\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}$$
### Ideal Camera Projection
$$ \lambda \begin{bmatrix}x\\y\\1\end{bmatrix} =
\begin{bmatrix}f & 0 & 0 \\0 & f & 0 \\0 & 0 & 1 \end{bmatrix} \cdot
\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix} \cdot
\begin{bmatrix}R & T \\0 & 1\end{bmatrix} \cdot
\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}$$
$$ \lambda \textbf{x} = K_f\Pi_0\mathbf{X} = K_f\Pi_0g\mathbf{X}_0$$
+ Canonical projection matrix $\Pi_0$
### From meters to pixels
+ Same units for world frame and image frame, i.e. meter
+ Meaningless - images are measured in pixels
$$\begin{bmatrix} x_s\\y_s\end{bmatrix} =
\begin{bmatrix} s_x & 0\\ 0 & s_y\end{bmatrix} \cdot
\begin{bmatrix} x\\y\end{bmatrix} $$
+ Each pixel is $s_x\times s_y$ in metric units
+ Then we translate the origin
$$x' = x_s + o_x \quad y' = y_s + o_y $$
+ or in homogeneous co-ordinates
$$\begin{bmatrix} x'\\y'\end{bmatrix} =
\begin{bmatrix} s_x & 0 & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot
\begin{bmatrix} x\\y\\1\end{bmatrix} $$
+ Sometimes the pixels are not rectangular
$$\begin{bmatrix} x'\\y'\\1\end{bmatrix} =
\begin{bmatrix} s_x & s_\theta & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot
\begin{bmatrix} x\\y\\1\end{bmatrix} $$
## Summary
+ **Camera Intrinsic Parameters**
$$\lambda\begin{bmatrix} x'\\y'\\1\end{bmatrix} =
\begin{bmatrix} s_x & s_\theta & o_x \\ 0 & s_y & o_y \\ 0 & 0 & 1 \end{bmatrix} \cdot
\begin{bmatrix} f & 0 & 0 \\ 0 & f & 0 \\ 0 & 0 & 1 \end{bmatrix} \cdot
\begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \cdot
\begin{bmatrix} X\\Y\\Z\\1\end{bmatrix} $$
+ Calibration is not a straight-forward matrix operation
+ It is not an operation on the luminance values of the pixels
+ Instead
1. The pixels are moved around in the plane, off the orignal grid
2. Then we need to resample the image to get a new set of pixel
in the original grid
## Distortion
### Calibration
+ Camera Calibration as described above
is a transformation in the *ideal* model
+ In addition we need calibration to compensate for distortion
= imperfection in the camera
+ Two types
+ Radial Distortion - typical for wide angle lenses
+ Tangential Distortion - when image plane and lens are not parallel
(ignored in the textbook)
### Radial Distortion
+ Artifact of wide angle lenses (wide field of view).
+ Simplest effective model:
$$
\begin{align}
x &= x_d(1 + a_1r^2 + a_2r^4) \\
y &= y_d(1 + a_1r^2 + a_2r^4) \\
r &= x_d^2+y_d^2
\end{align}
$$
+ $(x,y)$ are the true co-ordinate of some point
+ $(x_d,y_d)$ are the co-ordinates in the distorted image
+ This is easy to automate, so we can disregard distortion for analysis
+ OpenCV uses a sixth order model
$$
\begin{align}
x_d &= x(1 + a_1r^2 + a_2r^4 + a_3r^6) \\
y_d &= y(1 + a_1r^2 + a_2r^4 + a_3r^6)
\end{align}
$$
## Tangential Distortion
Not discussed in the text book
+ Occurs when the lens and image plane are not parallel
$$
\begin{align}
x_d &= x+ [2p_1xy+p_2(r^2+2x^2)]
y_d &= y+[p_1(r^2+2y^2)+2p_2xy]
\end{align}
$$
# Notes
We are going to do the calibration in practice.
+ OpenCV does not detect corners in all images and the tutorial recommends
running a loop searching for corners while you move the chessboard
+ [Sample Code](calibration.py)
+ You may have to tweak the delays so that you see what you are doing
in real time. I had to fiddle a lot to get this right.