What algorithm could be used to identify if images are the "same" or similar, regardless of size?

Question

TinEye, the "reverse image search engine", allows you to upload/link to an image and it is able to search through the billion images it has crawled and it will return links to images it has found that are the same image.

However, it isn't a naive checksum or anything related to that. It is often able to find both images of a higher resolution and lower resolution and larger and smaller size than the original image you supply. This is a good use for the service because I often find an image and want the highest resolution version of it possible.

Not only that, but I've had it find images of the same image set, where the people in the image are in a different position but the background largely stays the same.

What type of algorithm could TinEye be using that would allow it to compare an image with others of various sizes and compression ratios and yet still accurately figure out that they are the "same" image or set?

Solution 1

These algorithms are usually fingerprint-based. Fingerprint is a reasonably small data structure, something like a long hash code. However, the goals of fingerprint function are opposite to the goals of hash function. A good hash function should generate very different codes for very similar (but not equal) objects. The fingerprint function should, on contrary, generate the same fingerprint for similar images.

Just to give you an example, this is a (not particularly good) fingerprint function: resize the picture to 32x32 square, normalize and and quantize the colors, reducing the number of colors to something like 256. Then, you have 1024-byte fingerprint for the image. Just keep a table of fingerprint => [list of image URLs]. When you need to look images similar to a given image, just calculate its fingerprint value and find the corresponding image list. Easy.

What is not easy - to be useful in practice, the fingerprint function needs to be robust against crops, affine transforms, contrast changes, etc. Construction of good fingerprint functions is a separate research topic. Quite often they are hand-tuned and uses a lot of heuristics (i.e. use the knowledge about typical photo contents, about image format / additional data in EXIF, etc.)

Another variation is to use more than one fingerprint function, try to apply each of them and combine the results. Actually, it's similar to finding similar texts. Just instead of "bag of words" the image similarity search uses a "bag of fingerprints" and finds how many elements from one bag are the same as elements from another bag. How to make this search efficient is another topic.

Now, regarding the articles/papers. I couldn't find a good article that would give an overview of different methods. Most of the public articles I know discuss specific improvement to specific methods. I could recommend to check these:

"Content Fingerprinting Using Wavelets". This article is about audio fingerprinting using wavelets, but the same method can be adapted for image fingerprinting.

PERMUTATION GROUPING: INTELLIGENT HASH FUNCTION DESIGN FOR AUDIO & IMAGE RETRIEVAL. Info on Locality-Sensitive Hashes.

Bundling Features for Large Scale Partial-Duplicate Web Image Search. A very good article, talks about SIFT and bundling features for efficiency. It also has a nice bibliography at the end

Solution 2

The creator of the FotoForensics site posted this blog post on this topic, it was very useful to me, and showed algorithms that may be good enough for you and that require a lot less work than wavelets and feature extraction.

http://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html

aHash (also called Average Hash or Mean Hash). This approach crushes the image into a grayscale 8x8 image and sets the 64 bits in the hash based on whether the pixel's value is greater than the average color for the image.

pHash (also called "Perceptive Hash"). This algorithm is similar to aHash but use a discrete cosine transform (DCT) and compares based on frequencies rather than color values.

dHash Like aHash and pHash, dHash is pretty simple to implement and is far more accurate than it has any right to be. As an implementation, dHash is nearly identical to aHash but it performs much better. While aHash focuses on average values and pHash evaluates frequency patterns, dHash tracks gradients.

Solution 3

It's probably based on improvements of feature extraction algorithms, taking advantage of features which are scale invariant.

Take a look at

• Feature extraction

• SIFT, other site

or, if you are REALLY interested, you can shell out some 70 bucks (or at least look at the Google preview) for

• Feature Extraction & Image Processing

Solution 4

http://tineye.com/faq#how

Based on this, Igor Krivokon's answer seems to be on the mark.

Solution 5

The Hough Transform is a very old feature extraction algorithm, that you mind find interesting. I doubt it's what tinyeye uses, but it's a good, simple starting place for learning about feature extraction.

There are also slides to a neat talk from some University of Toronto folks about their work at astrometry.net. They developed an algorithm for matching telescoping images of the night sky to locations in star catalogs in order to identify the features in the image. It's a more specific problem than what tinyeye tries to solve, but I'd expect that a lot of the basic ideas that they talk about are applicable to the more general problem.

View more solutions

Comments

[Shoaib]

Interesting link... spelling police: I believe the site is actually called "TinEye", not "TinyEye". :)

[Igor Krivokon]

That's an interesting blog post, but I wouldn't take the algorithm used as an advice on the topic.

[Dennis Kioko]

@Igor, good answer! Do you have any links or resources you can provide on fingerprint algorithms other than the one you mentioned in your post?

[Vinko Vrsalovic]

+1. Do you have any reference to relevant papers? I've found a few but I don't know how good they are.

[Igor Krivokon]

@Vinko: actually, the book that you mentioned seems to be good (but I haven't read it).

[Hannes Ovrén]

Fourier Transforms would most likely be useless since natual images (i.e. photos) basically have the same frequency content (i.e. same magnitude, the phase differs). The rest sounds reasonable though.

[jilles de wit]

yep, SIFT, under heavy patent protection though, be aware of that for commercial development.

[Ankit Roy]

Why the downvote? The question was "What algorithm does TinEye use?" I just mentioned that the TinEye FAQ backs up Igor's answer.

[2vision2]

stackoverflow.com/questions/13507556/…

[Weihang Jian]

SSIM can be more accurate but it can't be indexed in database, unless you build a very huge table to store each SSIM value between each pair of images.

[Camilo Martin]

@JianWeihang You can still have the best of both worlds by using an algorithm that makes a very rough fingerprint of the image (like storing just an edge-detection bitmap in very low res), then calculating the SSIM of pairs that have the same fingerprint (or "nearly the same", which is a problem in and of itself). You don't even need to calculate the SSIM against every pair, just one of the ones that are "the same". But then again, it's a hell of a problem and I admire the fact Google and others work with it at such a scale.

[Dmitry Petrov]

In addition to theses hashes whash() was implemented in this library (2.0 version) - the same as pHash but it based on Discrete Wavelet Transformation (DWT) fullstackml.com/2016/07/02/wavelet-image-hash-in-python