Features Matching II & Homographies

This project implements feature generation and matching using OpenCV library. We observe two classes of object types: The first is solid cube shaped object such as a cereal box that is ideal for homography transforms and post processes. The second is deformable object such as a bag of chips that can be physically distored between two images and homographies and post processing may fail on them. In this project we observe the usage of optical flow on the 2nd category to check viability of estimating such distortions. I find that the optical flow optimization methodpost homography is robust to deformable objects and provides more complex mappings from source to destination images which had been a significant limitation of homographies alone.

Feature generation with SIFT

Item category 1: Solid Box

Item category 2: Deformable Bags

Feature matching with FLANN based KNN Matcher

Homography estimation and Transform

Below shows the source and the destination images rescaled to destination sizes with source images transformed to the perspectives of the destination image. The third image shows difference in pixels between the two. Note large difference in the 2nd category item.

Normalized Correlation Score: 0.977

Normalized Correlation Score: 0.742

The normalized correlation score shows us the similarity between source and destination images when overlaid on top of each other. Note that the score is way higher 0.977 for category 1 item vs category 2 item with score of 0.742 only. This difference can also be observed with the difference image shown in the third column above.

Optical Flow to visualize disortions

The usage of Optical flow provides vectors to the deformation in the destination object with respect to source. Note minimal flow in the first category, vs large flow in the second category as expected.

Using optical flow, we track the motion of each feature point from source to the destination image. We then remap the source image to destination using SKImages Piecewise Affine transform function. The result is a significant increase in the correlation score!

In the images below, left is source image transformed post optical flow. The middle shows alignment when overlayed. The third shows difference in pixels

Normalized Correlation Score is 0.982

Normalized Correlation Score has increased to 0.806 from 0.742

Another Item of Category 2

Source and Destination images.

Homography Transform. Normalized Correlation Score is 0.892

Optical flow vectors

Optical Flow based Transform. Normalized Correlation Score has increase to 0.93

Leave a Reply

Your email address will not be published. Required fields are marked *