---
title: Tracking Features
categories: lecture
---

- aperture problem [page 78]

# Briefing

## Corners

![Universitetsområdet i Ålesund](Images/ntnuaes1.jpg)

![Universitetsområdet i Ålesund (ny vinkel)](Images/ntnuaes2.jpg)

+ What are distinctive points in the image?
+ Distinctive points can (to some extent) be matched in two different images.

[More Images](https://www.flickr.com/photos/ntnu-trondheim/collections/72157632165205007/)

## Corner Correspondence

+ Two images of the same scene 
  $I_1,I_2: \Omega\subset\mathbb{R}^2\to\mathbb{R}_+ ; \mathbf{x}\mapsto I_1(\mathbf{x}),I_2(\mathbf{x})$
+ Different in general 

*Why are they different?*

+ Firstly - different colour in different directions
    + Lambertian assumption
+ Secondly - noise
    + let's assume this is insignificant
+ Thirdly - different image points $\mathbf{x}_1,\mathbf{x}_2$
  correspond to the same 3D point $p$

## Brightness Constancy Constraint

+  Suppose we photograph empty space except for a single point $p$
    - *Brightness Constancy Constraint*

$$I_1(\mathbf{x}_1) = I_2(\mathbf{x}_2) \sim \mathcal{R}(p)$$

+  Simple dislocation from $\mathbf{x}_1$ to $\mathbf{x}_2$ 
+  Motion: $h: \mathbf{x}_1\mapsto\mathbf{x}_2$ so that
   $$\forall
     \mathbf{x}_1\in\Omega\cap h^{-1}(\Omega)\subset\mathbb{R}^{2},
     \;I_1(\mathbf{x}_1) = I_2(h(\mathbf{x}_1))$$
      


### Motion Models

+ **Translational Motion Model:**
  $$h(\mathbf{x}_1) = \mathbf{x}_1 + \mathbf{\Delta x}$$
+ **Affine Motion Model:**
  $$h(\mathbf{x}_1) = A\mathbf{x}_1 + \mathbf{d}$$
+ **Projective Motion Model:**
  $$h(\mathbf{x}_1) = H\mathbf{x}_1$$
  where
  $H\in\mathbb{R}^{3\times3}$ is defined up to a scalar factor.

### Intencity Transformation

+ Need to accept changes to the intencity

$$I_1(\mathbf{x}_1) = I_2(h(\mathbf{x}_1)) + n(h(\mathbf{x}_1))$$

+ Occlusions
+ Non-Lambertian reflection
+ Taken at different time?  Different ambient light?

## Feature Tracking

+ Estimator
  $$\hat h = \arg\min_h\sum_{\tilde{\mathbf{x}}\in W(\mathbf{x})}
    ||I_1(\tilde{\mathbf{x}})-I_2(h(\tilde{\mathbf{x}}))||^2$$

+ The window, or aperture, $W(\vec{x})$
+ Choose $h$ from a family of functions, parameterised by $\alpha$
    + translational: $\alpha=\Delta\mathbf{x}$
    + affine: $\alpha=\{A,\mathbf{d}\}$
+ **Aperture problem:** cannot distinguish points on a blank wall

### Infinitesimal Model

+ Consider simple translational model 
  $$I_1(\textbf{x})= I_2(h(\textbf{x}))= I_2(\textbf{x}+\Delta\textbf{x})$$
+ Consider infitesimally small $\Delta\textbf x$
+ Model on a time axis 
    + two images taken infinitesimally close in time
    + ... under motion

1. First write $\mathbf{\Delta x} = \mathbf{u}dt$$, and
  rewrite the brightness constancy
  $$I(\mathbf{x}(t),t) = I(\mathbf{x}(t)+\mathbf{u}dt,t+dt)$$
2.  Apply Taylor series expansion and ignore higher-order terms
  $$\nabla I(\mathbf{x}(t),t)^\mathrm{T}\mathrm{u}dt + I_t(\mathbf{x}(t),t)dt = 0$$
  where
  $$\nabla I(\mathbf{x},t) = \begin{bmatrix} I_x(\mathbf{x},t)\\ I_y(\mathbf{x},t) \end{bmatrix}
= \begin{bmatrix}\frac{\partial I}{\partial x}(\mathbf{x},t)\\ \frac{\partial I}{\partial y}(\mathbf{x},t) \end{bmatrix}
\in\mathbb{R}^2$$
  and
  $$I_t(\mathbf{x},t) = \frac{\partial I(\mathbf{x},t)}{\partial t}\in \mathbb{R}$$
3. Simplify
  $$\nabla I(\mathbf{x}(t),t)^\mathrm{T}\mathrm{u} + I_t(\mathbf{x}(t),t) = 0$$

+ *Brightness Constancy Constraint* for the simplest possible continuous model
+ Two applications
    - optical flow: fix a position $\mathbf x$ and consider particles passing through
    - feature tracking: fix a particle $x(t)$ an track it through space 

### Solving for $\textbf{u}$

+ Consider the equation
  $$\nabla I^\mathrm{T}\mathrm{u} + I_t = 0$$
+ There are infititly many solutions, due to the *aperture problem*
    + We can solve for the component in the direction of the gradient though

1. Scalar projection of $\mathbf u$ onto $\nabla I$.
  $$\frac{\nabla I^\mathrm{T}\mathrm{u}}{||\nabla I||} = - \frac{I_t}{||\nabla I||} $$
2. Multiplying by $\nabla I/||\nabla I||$, we get the vector projection:
  $$\mathbf u_n =
    \frac{\nabla I^\mathrm{T}\mathrm{u}}{||\nabla I||}\cdot\frac{\nabla I}{||\nabla I||} =
    - \frac{I_t}{||\nabla I||}\cdot\frac{\nabla I}{||\nabla I||} $$

### Least squared errors estimate

+ Integrate over a window with sufficient texture
    + allows us to estimate $u$ in two dimensions
    + too many approximations for exact solution 
+ Minimise the sum of squared errors:
  $$E_b(\mathbf{u}) = \sum_{W(\mathbf{x}) [\nabla I^T(\tilde{\mathbf{x}}(\mathbf{u}(\mathbf{x})+I_t(\tilde{\mathbf{x}},t}]^2$$