This project implements feature generation and matching using OpenCV library. We observe two classes of object types: The first is solid cube shaped object such as a cereal box that is ideal for homography transforms and post processes. The second is deformable object such as a bag of chips that can be physically distored between two images and homographies and post processing may fail on them. In this project we observe the usage of optical flow on the 2nd category to check viability of estimating such distortions. I find that the optical flow optimization methodpost homography is robust to deformable objects and provides more complex mappings from source to destination images which had been a significant limitation of homographies alone.
Feature generation with SIFT
Item category 1: Solid Box
Item category 2: Deformable Bags
Feature matching with FLANN based KNN Matcher
Homography estimation and Transform
Below shows the source and the destination images rescaled to destination sizes with source images transformed to the perspectives of the destination image. The third image shows difference in pixels between the two. Note large difference in the 2nd category item.
Normalized Correlation Score: 0.977
Normalized Correlation Score: 0.742
The normalized correlation score shows us the similarity between source and destination images when overlaid on top of each other. Note that the score is way higher 0.977 for category 1 item vs category 2 item with score of 0.742 only. This difference can also be observed with the difference image shown in the third column above.
Optical Flow to visualize disortions
The usage of Optical flow provides vectors to the deformation in the destination object with respect to source. Note minimal flow in the first category, vs large flow in the second category as expected.
Using optical flow, we track the motion of each feature point from source to the destination image. We then remap the source image to destination using SKImages Piecewise Affine transform function. The result is a significant increase in the correlation score!
In the images below, left is source image transformed post optical flow. The middle shows alignment when overlayed. The third shows difference in pixels
Normalized Correlation Score is 0.982
Normalized Correlation Score has increased to 0.806 from 0.742
Another Item of Category 2
Source and Destination images.
Homography Transform. Normalized Correlation Score is 0.892
Optical flow vectors
Optical Flow based Transform. Normalized Correlation Score has increase to 0.93
