Revision 8164e8cf67943de5a5345e689fe39fc822d02fdc (click the page title to view the current version)
Changes from 8164e8cf67943de5a5345e689fe39fc822d02fdc to 5a65b982b511ad51fa4b3a9cc4c73b0b9ba4eff4
---
title: SIFT and Feature Matching
categories: session
---
# Briefing
In feature tracking, we use the time derivative to track
each feature point once detected.
In feature matching, we compare images which are not necessarily
related on the time axis. In other words, we need to compare
features directly, and pair corresponding features in two independent
images.
Two identify features, we use feature descriptors.
One of the most popular descriptors is called SIFT -
Scale Invariant Feature Transform.
Unfortunately, this is not described in the textbook.
For details, one can refer to
[Szeliski's](https://szeliski.org/Book/) textbook,
which is currently available as drafts for the second edition.
SIFT is described on page 435ff in the version of 30 September this year..
The principle used by SIFT is to gather statistics from the neighbourhood
around the feature point, and use this as an identifier.
# Exercises
In the exercise, you should use the OpenCV implementation of SIFT, and
test feature matching between pairs of imnages.
Start by finding a pair of images.
There has to be an overlap (30-40%) where you expect to find corners in both images.
You can of course use two frames from the video you used in the previous exercise.
## Exercise 1
For one of the images above, detect and draw corners as you have done earlier with the harris corner detector.<br>
Repeat using the SIFT detector implemented in opencv, e.g.:
```python
sift = cv.SIFT_create()
kps = sift.detect(img_gray)
kps, desc = sift.compute(img_gray, kps)
```
Compare the results, how does the keypoints correlate?
Use one of the two images in this exercise.
We are going to compare the results of the Harris detector with the SIFT
detector.
Opencv has the function `cv.drawKeypoints` that can be used to draw keypoints from the SIFT detector.<br>
This function will also visualize the "strength" of the keypoint with a larger circle,
does the strength affect the correlation?
1. Detect and draw corners as you have done earlier, using the harris corner detector.
2. Repeat using the SIFT detector implemented in opencv, e.g.:
```python
sift = cv.SIFT_create()
kps = sift.detect(img_gray)
kps, desc = sift.compute(img_gray, kps)
```
Compare the results, how does the keypoints correlate?
3. Opencv has the function `cv.drawKeypoints` that can be used to
draw keypoints from the SIFT detector.
Look up the documentation and test this funcion.
4. When the key points are drawn, the function will also visualize the
"strength" of the keypoint with a larger circle,
Does the strength affect the correlation?
## Exercise 2
Following is a code-snippet to print some information from one of the SIFT-keypoints:
```python
print(f"{kps[0].pt[0]}") # x: It is the position of the keypoint with respect to the x-axis
print(f"{kps[0].pt[1]}") # y: It is the position of the keypoint with respect to the y-axis
print(f"{kps[0].size}") # size: It is the diameter of the keypoint
print(f"{kps[0].angle}") # angle: It is the orientation of the keypoint
print(f"{kps[0].response}") # response: Also known as the strength of the keypoint, it is the keypoint detector response on the keypoint
print(f"{kps[0].octave}") # octave: It is the pyramid octave in which the keypoint has been detected
print(f"{desc[0]}") # keypoint-descriptor
```
Find a few keypoints and consider the meaning of the information from the snippet above.<br>
NB: The keypoints are not sorted, you can sort them by e.g. strength (descending) with the following code:
`kps_s, desc_s = zip(*sorted(list(zip(kps, desc)), key=lambda pt: pt[0].response, reverse=True))`
## Exercise 3
Apply SIFT to both images, where do you expect to find overlapping features?
Implement a brute-force feature matcher, using the euclidian distance between the keypoint descriptors. <br>
The euclidian distance can be calculated using the `cv.norm` with `normType=cv.NORM_L2`.<br>
More information on feature-matching can be found in chapter 7.1.3 of Szeliski’s textbook (linked above)
Combine/stack the two images horizontally and draw a line between the matching keypoints.
You can combine with e.g. `np.concatenate((img1, img2), axis=1)` (Only for gray-scale images of the same size)<br>
You can draw a line with `cv.line`
## Exercise 4
Where do you have True-Positives, False-Positives, False-Negatives and True-Negatives?<br>
(If you do not have any false-positives/Negatives, try with a more complex image set)<br>
## (Optional) Exercise 5
To remove false-positives we can use Lowe's ratio test, where we remove keypoints that have multiple possible matches.
From ( https://stackoverflow.com/questions/51197091/how-does-the-lowes-ratio-test-work ):<br>
"Short version: each keypoint of the first image is matched with a number of keypoints from the second image.
We keep the 2 best matches for each keypoint (best matches = the ones with the smallest distance measurement).
Lowe's test checks that the two distances are sufficiently different.
If they are not, then the keypoint is eliminated and will not be used for further calculations."
Implement this filtering function.
# Debrief
