Legacy/Exercises

Exercises
Additional Reading

Exercises

Setup

We will use scipy for a small part of the exercises, if you haven’t already, run ‘pip install scipy’.

We need an image to work with, you can either load an image from disk or capture a new image from the webcam with the below code.

# Imports
import cv2 as cv
import numpy as np

# Capture a single frame
vid = cv.VideoCapture(0)
image = None

# Capture frames until we click the space button, use that image
while True:
    _, frame = vid.read()

    # Display the resulting frame
    cv.imshow("frame", frame)

    # Click space to capture frame
    k = cv.waitKey(1)
    if k % 256 == 32:
        # SPACE pressed
        image = frame
        break

# After the loop release the cap object
vid.release()
# Destroy all the windows
cv.destroyAllWindows()

As we will be working with a lot of different images in this exercise, it is recommended to save it to disk, we can do that with

# Imports
from pathlib import Path

# First we create a path to an images directory
p = Path.cwd() / "images" # <--- current working directory + /images
if not p.is_dir():
    p.mkdir()

# Then we save the image to the directory with name "frame.jpg"
cv.imwrite(str(p / "frame.jpg"), image)

Now we convert the image to gray-scale

# Convert frame to grayscale
image_gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY)

# Save gray to images
cv.imwrite(str((p / "gray.jpg")), image_gray)

Exercise 1, Sobel/Derivative filter

The first exercise is to implement a Sobel-filter. Recall from the theory that we need to implement two 3x3 kernels to convolve with the original image.

This can be done using scipy.signal.convolve2d ( https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.convolve2d.html )

(Note: For a larger challenge you can also try implementing your own algorithm for the convolve function using numpy)

Code answers from here on will be collapsed, we recommend that you try to implement them yourself before reading an answer.

Hint (Click to expand)

Use scipy.signal.convolve2d(<image>, <filter>, boundary='symm', mode='same')

Solution (Click to expand)

  from scipy import signal

  sobel_x = np.array([[-1, 0, 1],
                      [-2, 0, 2],
                      [-1, 0, 1]])
  sobel_y = np.array([[-1, -2, -1],
                      [0, 0, 0],
                      [1, 2, 1]])

  grad_x = signal.convolve2d(image_gray, sobel_x, boundary='symm', mode='same')
  grad_y = signal.convolve2d(image_gray, sobel_y, boundary='symm', mode='same')

You should then show the images using cv.imshow or save using cv.imwrite, as we did earlier. Discuss the results.

You can compare the results of your implementation with the built in function cv.Sobel.

grad_x_cv = cv.Sobel(image_gray, cv.CV_32F, 1, 0, ksize=3)  # gradient along x axis,
grad_y_cv = cv.Sobel(image_gray, cv.CV_32F, 0, 1, ksize=3)  # gradient along y axis,

Using cv.Sobel, adjust the kernel-size to see how that changes the result.

Compute the magnitude and orientation of the derivatives using

magnitude = np.sqrt((grad_x ** 2) + (grad_y ** 2))
orientation = np.arctan2(grad_y, grad_x) * (180 / np.pi) % 180

Show/Save the images, and discuss.

Exercise 2, Eigenvalues and eigenvectors

TODO: ?

As this is not a trivial step, we will calculate the EigenVals and Vecs using cv.cornerEigenValsAndVecs, note that this function also applies a sobel-filter so we will call this function using the grayscale image.

bsize = 3  # the size of neighbourhood considered for corner detection
ksize = 3  # Aperture parameter of the Sobel derivative used.
eigenv = cv.cornerEigenValsAndVecs(image_gray, bsize, ksize)

The cornerEigenValsAndVecs algorithm will

Calculate the derivatives \(dI/dx\) and \(dI/dy\) using the sobel filter (as we did in exercise 1)
For each pixel \(p\), take a neighborhood \(S(p)\) of blockSize bsize*bsize, Calculate the covariation matrix \(M\) of the derivatives over the neighborhood as:

\[M = \begin{bmatrix} \sum_{S(p)}(dI/dx)^2 & \sum_{S(p)}(dI/dx)(dI/dy) \\ \sum_{S(p)}(dI/dx)(dI/dy) & \sum_{S(p)}(dI/dy)^2 \end{bmatrix} \]

After this the eigenvectors and eigenvalues of \(M\) are calculated and returned.

This should result in a \(h*w*6\) array with values (\(\lambda_1,\lambda_2,x_1,y_1,x_2,y_2\)) where

\(\lambda_1,\lambda_2\) are the non-sorted eigenvalues of \(M\)
\(x_1,y_1\) are the eigenvectors corresponding to \(\lambda_1\)
\(x_2,y_2\) are the eigenvectors corresponding to \(\lambda_2\)

Analyze? TODO: How can students learn from this exercise?

Exercise 3, Harris-criterion

Recall from the theory that we can calculate the harris-criterion with: \[\lambda_1 * \lambda_2 - k * (\lambda_1 + \lambda_2)^2\]

Create a new image of the same size as image_gray, and calculate the harris criterion for all pixels.

Solution (Click to expand)

A simple (naive) solution, is to create an empty image and iterate over all pixels like this:

(Note that this is a very computationally expensive solution, and takes a long time to complete)

  # Create an empty image of size image_gray.shape
  c_x = np.zeros(image_gray.shape, dtype=np.float32)

  # Define k, e.g. 0.04
  k = 0.04  # Harris free parameter

  # Iterate over image and set values
  for i in range(c_x.shape[0]):
    for j in range(c_x.shape[1]):
      lambda_1 = eigenv[i, j, 0]
      lambda_2 = eigenv[i, j, 1]
      c_x[i, j] = lambda_1 * lambda_2 - k * np.power((lambda_1 + lambda_2), 2)

Another, faster solution:

  # Define k, e.g. 0.04
  k = 0.04  # Harris free parameter

  # Extract eigen-values
  lambdas_1 = eigenv[:,:,0]
  lambdas_2 = eigenv[:,:,1]

  # Compute harris-criterion
  c_x = lambdas_1 * lambdas_2 - k * np.power((lambdas_1 + lambdas_2), 2)

Take a look at some of the values you calculated above and compare them to the original image (specifically where you expect the algorithm to detect corners). You can slice the image with the c_x[:,:] command, the syntax is [starting_row(including):ending_row(not including),starting_column(including):ending_column(not including)] , e.g. if you want to look at the top left corner around ~10% from the edge on a 480x480 image, you can use c_x[40:60, 40:60] or for the same but the bottom right corner, use e.g. c_x[-60:-40,-60:-40].

Exercise 4, Simple corner-detector

For the last exercise for corner-detection, you should draw a circle around all the corners you found on the original image/frame.

Start by picking a threshold T based on the information you found in the last exercise (or by using T = min_cx + (max_cx - min_cx) * 50 / 100) ). Now draw a circle around all pixels exceeding this value, using cv.circle.

Adjust the threshold T and try with different values to see how this changes the detection.

Solution (Click to expand)

A simple solution, is to iterate over all pixels and check compare the value against the threshold.

  min_cx = np.min(c_x)
  max_cx = np.max(c_x)
  T = min_cx + (max_cx - min_cx) * 50 / 100)
  f_circles = frame

  # Drawing a circle around corners (naive, slow)
  for i in range(f_circles.shape[0]):
    for j in range(f_circles.shape[1]):
      if c_x[i, j] > T:
        cv.circle(f_circles, (j, i), 5, (0, 0, 255), 2)

Another solution is to create a list of all pixels above the threshold before iterating over them:

  min_cx = np.min(c_x)
  max_cx = np.max(c_x)
  T = min_cx + (max_cx - min_cx) * 50 / 100)
  f_circles = frame

  mask = c_x > T
  corners = np.argwhere(mask).tolist()

  for corner in corners:
    cv.circle(f_circles, tuple(corner), 5, (0, 0, 255), 2)

Now do the same to an the original image based on the result from cv.cornerHarris and compare with your own result.

Using cv.cornerHarris, adjust block_size, aperture_size and the harris parameter (k), to see how they affect the result.

Exercise 5, SIFT

Exercise 6, Feature matching

Additional Reading

OpenCV/Python Tutorial
- Background: Understanding Features
- Harris Corner Detection
- Overview Feature Detection and Description
Ma 2004 Chapter 11.1-2.