Briefing

In feature tracking, we use the time derivative to track each feature point once detected.

In feature matching, we compare images which are not necessarily related on the time axis. In other words, we need to compare features directly, and pair corresponding features in two independent images.

Two identify features, we use feature descriptors. One of the most popular descriptors is called SIFT - Scale Invariant Feature Transform.

Unfortunately, this is not described in the textbook. For details, one can refer to Szeliski’s textbook, which is currently available as drafts for the second edition. SIFT is described on page 435ff in the version of 30 September this year..

The principle used by SIFT is to gather statistics from the neighbourhood around the feature point, and use this as an identifier.

Exercises

In the exercise, you should use the OpenCV implementation of SIFT, and test feature matching between pairs of imnages.

Start by finding a pair of images. There has to be an overlap (30-40%) where you expect to find corners in both images. You can of course use two frames from the video you used in the previous exercise.

Exercise 1

Use one of the two images in this exercise. We are going to compare the results of the Harris detector with the SIFT detector.

1. Detect and draw corners as you have done earlier, using the harris corner detector.

2. Repeat using the SIFT detector implemented in opencv, e.g.:

     sift = cv.SIFT_create()
     kps = sift.detect(img_gray)
   
     kps, desc = sift.compute(img_gray, kps)

   Compare the results, how does the keypoints correlate?
3. Opencv has the function cv.drawKeypoints that can be used to draw keypoints from the SIFT detector. Look up the documentation and test this funcion.

4. When the key points are drawn, the function will also visualize the “strength” of the keypoint with a larger circle, Does the strength affect the correlation?

Exercise 2

Following is a code-snippet to print some information from one of the SIFT-keypoints:

print(f"{kps[0].pt[0]}") # x: It is the position of the keypoint with respect to the x-axis
print(f"{kps[0].pt[1]}") # y: It is the position of the keypoint with respect to the y-axis
print(f"{kps[0].size}") # size: It is the diameter of the keypoint
print(f"{kps[0].angle}") # angle: It is the orientation of the keypoint
print(f"{kps[0].response}") # response: Also known as the strength of the keypoint, it is the keypoint detector response on the keypoint
print(f"{kps[0].octave}") # octave: It is the pyramid octave in which the keypoint has been detected
print(f"{desc[0]}") # keypoint-descriptor

Find a few keypoints and consider the meaning of the information from the snippet above.
NB: The keypoints are not sorted, you can sort them by e.g. strength (descending) with the following code:

kps_s, desc_s = zip(*sorted(list(zip(kps, desc)), key=lambda pt: pt[0].response, reverse=True))

Exercise 3

Apply SIFT to both images, where do you expect to find overlapping features?

Implement a brute-force feature matcher, using the euclidian distance between the keypoint descriptors.
The euclidian distance can be calculated using the cv.norm with normType=cv.NORM_L2.
More information on feature-matching can be found in chapter 7.1.3 of Szeliski’s textbook (linked above)

Combine/stack the two images horizontally and draw a line between the matching keypoints.

You can combine with e.g. np.concatenate((img1, img2), axis=1) (Only for gray-scale images of the same size)
You can draw a line with cv.line

Exercise 4

Where do you have True-Positives, False-Positives, False-Negatives and True-Negatives?
(If you do not have any false-positives/Negatives, try with a more complex image set)

(Optional) Exercise 5

To remove false-positives we can use Lowe’s ratio test, where we remove keypoints that have multiple possible matches.

From StackOverflow:
“Short version: each keypoint of the first image is matched with a number of keypoints from the second image. We keep the 2 best matches for each keypoint (best matches = the ones with the smallest distance measurement). Lowe’s test checks that the two distances are sufficiently different. If they are not, then the keypoint is eliminated and will not be used for further calculations.”

Implement this filtering function.

Debrief