Revision 902402841c9aa4b32baef28e1603f6dd6d6fceb0 (click the page title to view the current version)

Corner Detection

Changes from 902402841c9aa4b32baef28e1603f6dd6d6fceb0 to 3efe08427e796145439fd0c0d4b7ea47311e32b5

---
title: Corner Detection
categories: session
---

# Briefing

## Corners and Feature Points

## Differentiation

## Harris Feature Detector

# Exercises

## Setup

We will use scipy for a small part of the exercises,
if you haven't already, run 'pip install scipy'.

We need an image to work with, you can either load an image from disk or
capture a new image from the webcam with the below code.

```python
# Imports
import cv2 as cv
import numpy as np

# Capture a single frame
vid = cv.VideoCapture(0)
image = None

# Capture frames until we click the space button, use that image
while True:
    _, frame = vid.read()

    # Display the resulting frame
    cv.imshow("frame", frame)

    # Click space to capture frame
    k = cv.waitKey(1)
    if k % 256 == 32:
        # SPACE pressed
        image = frame
        break

# After the loop release the cap object
vid.release()
# Destroy all the windows
cv.destroyAllWindows()
```

As we will be working with a lot of different images in this exercise,
it is recommended to save it to disk, we can do that with

```python
# Imports
from pathlib import Path

# First we create a path to an images directory
p = Path.cwd() / "images" # <--- current working directory + /images
if not p.is_dir():
    p.mkdir()

# Then we save the image to the directory with name "frame.jpg"
cv.imwrite(str(p / "frame.jpg"), image)
```

Now we convert the image to gray-scale

```python
# Convert frame to grayscale
image_gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY)

# Save gray to images
cv.imwrite(str((p / "gray.jpg")), image_gray)
```

## Exercise 1

The first exercise is to implement a Sobel-filter.
Recall from the theory that we need to implement two 3x3 kernels to convolve with the original image.

This can be done using scipy.signal.convolve2d ( https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.convolve2d.html )

(Note: For a larger challenge you can also try implementing your own algorithm for the convolve function using numpy)

Code answers from here on will be collapsed, we recommend that you try to
implement them yourself before reading an answer.

<details>
  <summary>Hint (Click to expand)</summary>
Note: Use signal.convolve2d(`<image>`, `<filter>`, boundary='symm', mode='same')
</details>
&nbsp;

<details>
  <summary>Solution (Click to expand)</summary>

  ```python
  from scipy import signal

  sobel_x = np.array([[-1, 0, 1],
                      [-2, 0, 2],
                      [-1, 0, 1]])
  sobel_y = np.array([[-1, -2, -1],
                      [0, 0, 0],
                      [1, 2, 1]])

  grad_x = signal.convolve2d(image_gray, sobel_x, boundary='symm', mode='same')
  grad_y = signal.convolve2d(image_gray, sobel_y, boundary='symm', mode='same')
  ```
</details>
&nbsp;

You should then show the images using cv.imshow or save using cv.imwrite, as we did earlier.
Discuss the results.

You can compare the results of your implementation with the built in function
`cv.Sobel` (the `cv.CV_64F`

```python
grad_x_cv = cv.Sobel(image_gray, cv.CV_32F, 1, 0, ksize=3)  # gradient along x axis,
grad_y_cv = cv.Sobel(image_gray, cv.CV_32F, 0, 1, ksize=3)  # gradient along y axis,
```

Compute the magnitude and orientation of the derivatives using
```python
magnitude = np.sqrt((grad_x ** 2) + (grad_y ** 2))
orientation = np.arctan2(grad_y, grad_x) * (180 / np.pi) % 180
```

Show/Save the images, and discuss.

## Exercise 2

TODO: ?

As this is not a trivial step, we will calculate the EigenVals and Vecs using
`cv.cornerEigenValsAndVecs`, note that this function also calculates applies a sobel-filter
so we will call this function using the grayscale image.

```python
bsize = 3  # the size of neighbourhood considered for corner detection
ksize = 3  # Aperture parameter of the Sobel derivative used.
eigenv = cv.cornerEigenValsAndVecs(image_gray, bsize, ksize)
```

The cornerEigenValsAndVecs algorithm will

1. Calculate the derivatives $dI/dx$ and $dI/dy$ using the sobel filter (as we did in exercise 1)
2. For each pixel $p$, take a neighborhood $S(p)$ of blockSize `bsize*bsize`,
Calculate the covariation matrix $M$ of the derivatives over the neighborhood as:

$$M = \begin{bmatrix}
        \sum_{S(p)}(dI/dx)^2 & \sum_{S(p)}(dI/dx)(dI/dy) \\
        \sum_{S(p)}(dI/dx)(dI/dy) & \sum_{S(p)}(dI/dy)^2
      \end{bmatrix}
$$

After this the eigenvectors and eigenvalues of M are calculated and returned.

This should result in a `h*w*6` array with values ($\lambda_1$,$\lambda_2$,x1,y1,x2,y2) where
This should result in a `h*w*6` array with values ($\lambda_1,\lambda_2,x_1,y_1,x_2,y_2$) where

* $\lambda_1$,$\lambda_2$ are the non-sorted eigenvalues of M
* x1,y1 are the eigenvectors corresponding to $\lambda_1$
* x2,y2 are the eigenvectors corresponding to $\lambda_2$
* $x_1,y_1$ are the eigenvectors corresponding to $\lambda_1$
* $x_2,y_2$ are the eigenvectors corresponding to $\lambda_2$

Analyze? TODO: How can students learn from this exercise?

Exercise 3
## Exercise 3

Recall from the theory that we can calculate the harris-criterion with:
$$\lambda_1 * \lambda_2 - k * (\lambda_1 + \lambda_1)^2$$

# Debrief