Revision e476811c8daaa9cf99fca9658dae438752b68494 (click the page title to view the current version)
Changes from e476811c8daaa9cf99fca9658dae438752b68494 to 655bc5b96e93445c0264e099a150494e5ca3342b
---
title: SIFT and Feature Matching
categories: session
---
# Briefing
In feature tracking, we use the time derivative to track
each feature point once detected.
In feature matching, we compare images which are not necessarily
related on the time axis. In other words, we need to compare
features directly, and pair corresponding features in two independent
images.
Two identify features, we use feature descriptors.
One of the most popular descriptors is called SIFT -
Scale Invariant Feature Transform.
Unfortunately, this is not described in the textbook.
For details, one can refer to
[Szeliski's](https://szeliski.org/Book/) textbook,
which is currently available as drafts for the second edition.
SIFT is described on page 435ff in the version of 30 September this year..
The principle used by SIFT is to gather statistics from the neighbourhood
around the feature point, and use this as an identifier.
# Exercises
In the exercise, you should use the OpenCV implementation of SIFT, and
test feature matching between pairs of imnages.
Start by finding a pair of images.
Start by finding one or more pairs of images.
There has to be an overlap (30-40%) where you expect to find corners in both images.
You can of course use two frames from the video you used in the previous exercise.
It may be a good idea to test different pairs, to learn what can and cannot be
matched.
## Exercise 1
Use one of the two images in this exercise.
We are going to compare the results of the Harris detector with the SIFT
detector.
1. Detect and draw corners as you have done earlier, using the harris corner detector.
2. Repeat using the SIFT detector implemented in opencv, e.g.:
```python
sift = cv.SIFT_create()
kps = sift.detect(img_gray)
kps, desc = sift.compute(img_gray, kps)
```
Compare the results, how does the keypoints correlate?
3. Opencv has the function `cv.drawKeypoints` that can be used to
draw keypoints from the SIFT detector.
Look up the documentation and test this funcion.
4. When the key points are drawn, the function will also visualize the
"strength" of the keypoint with a larger circle,
Does the strength affect the correlation?
## Exercise 2
Following is a code-snippet to print some information from one of the SIFT-keypoints:
```python
print(f"{kps[0].pt[0]}") # x: It is the position of the keypoint with respect to the x-axis
print(f"{kps[0].pt[1]}") # y: It is the position of the keypoint with respect to the y-axis
print(f"{kps[0].size}") # size: It is the diameter of the keypoint
print(f"{kps[0].angle}") # angle: It is the orientation of the keypoint
print(f"{kps[0].response}") # response: Also known as the strength of the keypoint, it is the keypoint detector response on the keypoint
print(f"{kps[0].octave}") # octave: It is the pyramid octave in which the keypoint has been detected
print(f"{desc[0]}") # keypoint-descriptor
```
Find a few keypoints and consider the meaning of the information from the snippet above.<br>
NB: The keypoints are not sorted, you can sort them by e.g. strength (descending) with the following code:
`kps_s, desc_s = zip(*sorted(list(zip(kps, desc)), key=lambda pt: pt[0].response, reverse=True))`
## Exercise 3
Apply SIFT to both images, where do you expect to find overlapping features?
Implement a brute-force feature matcher, using the euclidian distance between the keypoint descriptors. <br>
The euclidian distance can be calculated using the `cv.norm` with `normType=cv.NORM_L2`.<br>
More information on feature-matching can be found in chapter 7.1.3 of Szeliski’s textbook (linked above)
Implement a brute-force feature matcher,
using the Euclidian distance between the keypoint descriptors.
In other words, we match a point (descriptor) $X_1$ in image 1 with a
point (descriptor) $X_2$ in image 2 if $X_2$, if the Euclidean distance
$||X_1-X_2||$ is smaller than $||X_1-X||$ for any other point $X$ in Image 2.
The Euclidian distance can be calculated using the `cv.norm` with
`normType=cv.NORM_L2`.<br>
More information on feature-matching can be found in chapter 7.1.3
of [Szeliski’s textbook](https://szeliski.org/Book/) textbook,
Combine/stack the two images horizontally and draw a line between the matching keypoints.
You can combine with e.g. `np.concatenate((img1, img2), axis=1)` (Only for gray-scale images of the same size)<br>
You can draw a line with `cv.line`
+ You can combine with e.g. `np.concatenate((img1, img2), axis=1)` (Only for gray-scale images of the same size)
+ You can draw a line with `cv.line`
+ Use the OpenCV documentation for details.
## Exercise 4
Where do you have True-Positives, False-Positives, False-Negatives and True-Negatives?<br>
(If you do not have any false-positives/Negatives, try with a more complex image set)<br>
## (Optional) Exercise 5
To remove false-positives we can use Lowe's ratio test, where we remove keypoints that have multiple possible matches.
From [StackOverflow](https://stackoverflow.com/questions/51197091/how-does-the-lowes-ratio-test-work):
<br>
"Short version: each keypoint of the first image is matched with a number of keypoints from the second image.
We keep the 2 best matches for each keypoint (best matches = the ones with the smallest distance measurement).
Lowe's test checks that the two distances are sufficiently different.
If they are not, then the keypoint is eliminated and will not be used for further calculations."
Implement this filtering function.
# Debrief
