Revision a517c117229d450905371c69681704e5532ee07d (click the page title to view the current version)

SIFT

Changes from a517c117229d450905371c69681704e5532ee07d to 4ff6659f7c3bfae94a7ff0c1405db1710454893f

---
title: SIFT and Feature Matching
categories: session
---

# Briefing

In feature tracking, we use the time derivative to track
each feature point once detected.

In feature matching, we compare images which are not necessarily 
related on the time axis.  In other words, we need to compare
features directly, and pair corresponding features in two independent
images.

Two identify features, we use feature descriptors.
One of the most popular descriptors is called SIFT -
Scale Invariant Feature Transform.

Unfortunately, this is not described in the textbook.
For details, one can refer to 
[Szeliski's](https://szeliski.org/Book/) textbook,
which is currently available as drafts for the second edition.
SIFT is described on page 435ff in the version of 30 September this year..

The principle used by SIFT is to gather statistics from the neighbourhood
around the feature point, and use this as an identifier.

# Exercises

In the exercise, you should use the OpenCV implementation of SIFT.

In this exercise set you will work with a set of two images,
the images should have some overlap (30-40%) where you expect to find corners in both images. <br>
Start by finding or creating such a set of images (e.g. by using two frames from the previous exercise set)

## Exercise 1

For one of the images above, detect and draw corners as you have done earlier with the harris corner detector.<br>
Repeat using the SIFT detector implemented in opencv, e.g.:

```python
  sift = cv.SIFT_create()
  kps = sift.detect(img_gray)

  kps, desc = sift.compute(img_gray, kps)
```

Compare the results, how does the keypoints correlate?

Opencv has the function `cv.drawKeypoints` that can be used to draw keypoints from the SIFT detector.<br>
This function will also visualize the "strength" of the keypoint with a larger circle,
does the strength affect the correlation?

## Exercise 2

Following is a code-snippet to print some information from one of the SIFT-keypoints:

```python
print(f"{kps[0].pt[0]}") # x: It is the position of the keypoint with respect to the x-axis
print(f"{kps[0].pt[1]}") # y: It is the position of the keypoint with respect to the y-axis
print(f"{kps[0].size}") # size: It is the diameter of the keypoint
print(f"{kps[0].angle}") # angle: It is the orientation of the keypoint
print(f"{kps[0].response}") # response: Also known as the strength of the keypoint, it is the keypoint detector response on the keypoint
print(f"{kps[0].octave}") # octave: It is the pyramid octave in which the keypoint has been detected
print(f"{desc[0]}") # keypoint-descriptor
```

Find a few keypoints and consider the meaning of the information from the snippet above.<br>
NB: The keypoints are not sorted, you can sort them by e.g. strength (descending) with the following code:

`kps_s, desc_s = zip(*sorted(list(zip(kps, desc)), key=lambda pt: pt[0].response, reverse=True))`

## Exercise 3

Apply SIFT to both images, where do you expect to find overlapping features?

Implement a brute-force feature matcher, using the euclidian distance between the keypoint descriptors. <br>
The euclidian distance can be calculated using the `cv.norm` with `normType=cv.NORM_L2`.<br>
More information on feature-matching can be found in chapter 7.1.3 of Szeliski’s textbook (linked above)

Combine/stack the two images horizontally and draw a line between the matching keypoints.

You can combine with e.g. `np.concatenate((img1, img2), axis=1)` (Only for gray-scale images of the same size)<br>
You can draw a line with `cv.line`

## Exercise 4

Where do you have True-Positives, False-Positives, False-Negatives and True-Negatives?<br>
(If you do not have any false-positives/Negatives, try with a more complex image set)<br>

## (Optional) Exercise 5

To remove false-positives we can use Lowe's ratio test, where we remove keypoints that have multiple possible matches.

From ( https://stackoverflow.com/questions/51197091/how-does-the-lowes-ratio-test-work ):<br>
"Short version: each keypoint of the first image is matched with a number of keypoints from the second image.
We keep the 2 best matches for each keypoint (best matches = the ones with the smallest distance measurement).
Lowe's test checks that the two distances are sufficiently different.
If they are not, then the keypoint is eliminated and will not be used for further calculations."

Implement this filtering function.

# Debrief