• aperture problem [page 78]

Briefing

Corners

• What are distinctive points in the image?
• Distinctive points can (to some extent) be matched in two different images.

More Images

Corner Correspondence

• Two images of the same scene \(I_1,I_2: \Omega\subset\mathbb{R}^2\to\mathbb{R}_+ ; \mathbf{x}\mapsto I_1(\mathbf{x}),I_2(\mathbf{x})\)
• Different in general

Why are they different?

• Firstly - different colour in different directions
  • Lambertian assumption
• Secondly - noise
  • let’s assume this is insignificant
• Thirdly - same point in different positions
  • different image points \(\mathbf{x}_1,\mathbf{x}_2\) correspond to the same 3D point \(p\)

Note Two approaches: feature descriptors and motion modelling.

Today Motion modelling using differentiation, which is adequate for slow motion, say 2-3 pixels per frame.

Brightness Constancy Constraint

• Suppose we photograph empty space except for a single point \(p\)
  • Brightness Constancy Constraint

\[I_1(\mathbf{x}_1) = I_2(\mathbf{x}_2) \sim \mathcal{R}(p)\]

• Simple dislocation from \(\mathbf{x}_1\) to \(\mathbf{x}_2\)
• Motion: \(h: \mathbf{x}_1\mapsto\mathbf{x}_2\) so that \[\forall \mathbf{x}_1\in\Omega\cap h^{-1}(\Omega)\subset\mathbb{R}^{2}, \;I_1(\mathbf{x}_1) = I_2(h(\mathbf{x}_1))\]

Motion Models

• Translational Motion Model: \[h(\mathbf{x}_1) = \mathbf{x}_1 + \mathbf{\Delta x}\]
• Affine Motion Model: \[h(\mathbf{x}_1) = A\mathbf{x}_1 + \mathbf{d}\]
• Projective Motion Model: \[h(\mathbf{x}_1) = H\mathbf{x}_1\] where \(H\in\mathbb{R}^{3\times3}\) is defined up to a scalar factor.

Intencity Transformation

• Need to accept changes to the intencity

\[I_1(\mathbf{x}_1) = I_2(h(\mathbf{x}_1)) + n(h(\mathbf{x}_1))\]

• Occlusions
• Non-Lambertian reflection
• Taken at different time? Different ambient light?

Feature Tracking

• Estimator \[\hat h = \arg\min_h\sum_{\tilde{\mathbf{x}}\in W(\mathbf{x})} ||I_1(\tilde{\mathbf{x}})-I_2(h(\tilde{\mathbf{x}}))||^2\]

• The window, or aperture, \(W(\vec{x})\)
• Choose \(h\) from a family of functions, parameterised by \(\alpha\)
  • translational: \(\alpha=\Delta\mathbf{x}\)
  • affine: \(\alpha=\{A,\mathbf{d}\}\)

• Aperture problem: cannot distinguish points on a blank wall

Infinitesimal Model

• Consider simple translational model \[I_1(\textbf{x})= I_2(h(\textbf{x}))= I_2(\textbf{x}+\Delta\textbf{x})\]
• Consider infitesimally small \(\Delta\textbf x\)
• Model on a time axis
  • two images taken infinitesimally close in time
  • … under motion

1. First write \(\mathbf{\Delta x} = \mathbf{u}dt\)$, and rewrite the brightness constancy \[I(\mathbf{x}(t),t) = I(\mathbf{x}(t)+\mathbf{u}dt,t+dt)\]
2. Apply Taylor series expansion and ignore higher-order terms \[\nabla I(\mathbf{x}(t),t)^\mathrm{T}\mathrm{u}dt + I_t(\mathbf{x}(t),t)dt = 0\] where \[\nabla I(\mathbf{x},t) = \begin{bmatrix} I_x(\mathbf{x},t)\\ I_y(\mathbf{x},t) \end{bmatrix} = \begin{bmatrix}\frac{\partial I}{\partial x}(\mathbf{x},t)\\ \frac{\partial I}{\partial y}(\mathbf{x},t) \end{bmatrix} \in\mathbb{R}^2\] and \[I_t(\mathbf{x},t) = \frac{\partial I(\mathbf{x},t)}{\partial t}\in \mathbb{R}\]
3. Simplify \[\nabla I(\mathbf{x}(t),t)^\mathrm{T}\mathrm{u} + I_t(\mathbf{x}(t),t) = 0\]

• Brightness Constancy Constraint for the simplest possible continuous model
• Two applications
  • optical flow: fix a position \(\mathbf x\) and consider particles passing through
  • feature tracking: fix a particle \(x(t)\) an track it through space

Solving for \(\textbf{u}\)

• Consider the equation \[\nabla I^\mathrm{T}\mathrm{u} + I_t = 0\]
• There are infititly many solutions, due to the aperture problem
  • We can solve for the component in the direction of the gradient though

1. Scalar projection of \(\mathbf u\) onto \(\nabla I\). \[\frac{\nabla I^\mathrm{T}\mathrm{u}}{||\nabla I||} = - \frac{I_t}{||\nabla I||} \]
2. Multiplying by \(\nabla I/||\nabla I||\), we get the vector projection: \[\mathbf u_n = \frac{\nabla I^\mathrm{T}\mathrm{u}}{||\nabla I||}\cdot\frac{\nabla I}{||\nabla I||} = - \frac{I_t}{||\nabla I||}\cdot\frac{\nabla I}{||\nabla I||} \]

Least squared errors estimate

• Integrate over a window with sufficient texture
  • allows us to estimate \(u\) in two dimensions
  • too many approximations for exact solution
• Minimise the sum of squared errors: \[E_b(\mathbf{u}) = \sum_{W(\mathbf{x})} [\nabla I^T(\tilde{\mathbf{x}}(\mathbf{u}(\mathbf{x})+I_t(\tilde{\mathbf{x}},t)]^2\]
• Differentiate \[\nabla E_b(\mathbf{u}) = 2\sum_{W(\mathbf{x})} \nabla I [\nabla I^T\mathbf{u}+I_t]\]
• Spelling out the matrices, we have \[\nabla E_b(\mathbf{u}) = 2\sum_{W(\mathbf{x})} \bigg(\begin{bmatrix} I_x^2 & I_xI_y \\ I_xI_y & I_y^2\end{bmatrix}\mathbf{u} + \begin{bmatrix} I_xI_t \\ I_yI_t\end{bmatrix}\bigg)\]
• To minimise \(E_b\), the derivative should be zero \[0 = \begin{bmatrix} \sum I_x^2 & \sum I_xI_y \\ \sum I_xI_y & \sum I_y^2 \end{bmatrix}\mathbf{u} + \begin{bmatrix} \sum I_xI_t \\ \sum I_yI_t\end{bmatrix}\]
• We refer to the first matrix as \(G\), so that \[G\mathbf{u} + \mathbf{b} = 0\]
• If \(G\) is non-singular, we have \[\mathbf{u} = - G^{-1}\mathbf{b}\]
• This gives us the motion vector \(\mathbf{u}\)

Algorithm (4.1 of Ma 2004)

Compute \[G(\mathbf{x}) = \begin{bmatrix} \sum I_x^2 & \sum I_xI_y \\ \sum I_xI_y & \sum I_y^2 \end{bmatrix}\] and \[b(\mathbf{x},t) = \begin{bmatrix} \sum I_xI_t \\ \sum I_yI_t\end{bmatrix}\] at every pixel \(\mathbf{x}\).

• Feature tracking choose feature points \(x_1,x_2,\ldots\) where \(G(\mathbf{x})\) is invertible
• Optical flow choose points \(x_1,x_2,\ldots\) on a fixed grid

\[\mathbf{u}(x,t) = \begin{cases} - G^{-1}b&\quad\text{if defined}\\ 0&\quad\text{otherwise} \end{cases} \]

• Feature tracking at time \(t+1\) replace point \(x\) by \(x+\mathbf{u}(x,t)\)
• Optical flow at time \(t+1\) repeat operation at the same point \(x\)