Image similarity comparison

Question

I originally asked this question on cstheory.stackexchange.com but was suggested to move it to stats.stackexchange.com.

Is there an existing algorithm that returns to me a similarity metric between two bitmap images? By "similar", I mean a human would say these two images were altered from the same photograph. For example, the algorithm should say the following 3 images are the same (original, position shifted, shrunken).

Same

I don't need to detect warped or flipped images. I also don't need to detect if it's the same object in different orientations.

Different

I would like to use this algorithm to prevent spam on my website. I noticed that the spammers are too lazy to change their spam images. It's not limited to faces. I already know there's already many great facial recognition algorithms out there. The spam image could be anything from a URL to a soccer field to a naked body.

Solution 1

There is a discussion of image similarity algorithms at stack overflow. Since you don't need to detect warped or flipped images, the histogram approach may be sufficient providing the image crop isn't too severe.

Solution 2

You can use existing deep learning architectures like VGG to generate features from images and then use a similarity metric like cosine similarity to see if two images are essentially the same.

The whole pipeline is pretty easy to set up and you do not need to understand the neural network architecture (you can just treat it like a black box). Also, these features are pretty generic and can be applied to find similarity between any kind of objects, not just faces.

Here are a couple of blogs that walk you through the process. http://blog.ethanrosenthal.com/2016/12/05/recasketch-keras/ https://erikbern.com/2015/09/24/nearest-neighbor-methods-vector-models-part-1.html

Solution 3

Amazon has a new API called Rekognition which allows you to compare two images for facial similarity. The api returns a similarity percentage for each face with one another and the bounding boxes for each face.

Rekognition also includes an api for both Facial Analysis (returning the gender, approximate age, and other relevant facial details) and Object Scene Detection(returning tags of objects that are within in image).

Solution 4

One of the great technique to calculate similarity of images is "mean structural similarity".

import cv2
from skimage import compare_ssim


img = cv2.imread('img_1.png')
img_2 = cv2.imread('img_2.png')

print(compare_ssim(img, img_2))

Solution 5

Use https://github.com/Netflix/vmaf to compare the two sets of images.

First convert the images to yuv422p using ffmpeg and then run the test. Note the score difference. This can be used to tell if the image is similar or different. For this sample they both look quite similiar...

ffmpeg -i .\different-pose-1.jpg  -s 1920x1080 -pix_fmt yuv422p different-pose-1.yuv
ffmpeg -i .\different-pose-2.jpg  -s 1920x1080 -pix_fmt yuv422p different-pose-2.yuv
.\vmafossexec.exe yuv422p 1920 1080 different-pose-1.yuv different-pose-2.yuv vmaf_v0.6.1.pkl --ssim --ms-ssim --log-fmt json --log different.json
Start calculating VMAF score...
Exec FPS: 0.772885
VMAF score = 2.124272
SSIM score = 0.424488
MS-SSIM score = 0.415149

ffmpeg.exe -i .\same-pose-1.jpg  -s 1920x1080 -pix_fmt yuv422p same-pose-1.yuv
ffmpeg.exe -i .\same-pose-2.jpg  -s 1920x1080 -pix_fmt yuv422p same-pose-2.yuv
.\vmafossexec.exe yuv422p 1920 1080 same-pose-1.yuv same-pose-2.yuv vmaf_v0.6.1.pkl --ssim --ms-ssim --log-fmt json --log same.json
Start calculating VMAF score...
Exec FPS: 0.773098
VMAF score = 5.421821
SSIM score = 0.285583
MS-SSIM score = 0.400130

References How can I create a YUV422 frame from a JPEG or other image on Ubuntu

View more solutions

Comments

[pathikrit]

This gives me an excellent idea for a reverse-Turing test (I hate CAPTCHAs)

[meaning-matters]

Are there always solid black borders?

[Learning C]

I've been using rapidapi.com/apigeek/api/image-diff4 to check for duplicate uploads. You can pass it two URL to check if two image are similar.

[Sundeep Pidugu]

any references to VGG to generate features?

[Anvesh K]

This should help you github.com/kanvesh/DataMining/blob/master/…