Get crisp, clear thumbnails from MP4 with FFmpeg / ImageMagick

Question

My ultimate goal is to get meaningful snapshots from videos that are either 30 min or 1 hour long. "Meaningful" is a bit ambitious, so I have simplified my requirements.

The image should be crisp - not blurry.

Initially, I thought this meant getting a "keyframe". Since there are many keyframes, I decided to choose the keyframe closest to the third minute of the video, which was generally "meaningful" enough for me. I followed the advice at: FFmpeg command to find key frame closest to 3rd minute

But the problem is that these keyframes are often (not always) blurry. An example is:

I then tried, Meaningful thumbnails for a Video using FFmpeg which did help get more meaningful snapshots, but I still often (not always) got blurry frames like the above.

You will notice that this sort of image is essentially an overlap of 2 different scenes. Sometimes, however, I get images that work for me – like this:

The above image is not very meaningful, but it is crisp.

Ideally, I would like to FFmpeg not to return blurry frames. Alternatively, I would like to use a script to detect blurry frames and select the least blurry from say 5 frames. Does anyone know how to do this?

Solution

"Meaningful" is quite subjective but "blurry" is fairly objective and simple to detect.

I had a similar issue and after a little bit of researching ended up with the following algorithm:

• Determine video file length in seconds.

• Generate 10 png thumbnails using ffmpeg and scene change detecion with something like:

  ffmpeg -ss {$skip} -i {$input} -vf "select=gt(scene\,0.3)" -r 1 -frames:v 1 {$output}
  

On each iteration $skip is increased by 10% on the video length in seconds. The parameter 0.3 may not be most appropriate for you, but you can play arround with it. This solved the problem with the overlapping scenes and/or completely blurred images.

• Detect the edges of those 10 thumbnails and downscale them for better performance with ImageMagick:

  convert {$input} -thumbnail {$w}x{$h} -colorspace Gray -edge 1 {$path_to_downscaled_image}
  

I'm using -thumbnail to scale the image. Then I'm adding a black and white filter. Finally I do the edge detection with a radius of 1. This produces a black and white image only with the edges marked as white. This means that the more white the image has the more edges there are in it. You will get something like this:

• Identify the resulting black and white images with ImageMagick:

  identify -format '{$format}' {$path_to_downscaled_image}
  

What you should be interested is %[mean] and %[standard-deviation]. Play arround with those and see what suits you best. Just sort all the resulting images by "%[mean] + %[standard-deviation]" for example. Find the same image before we started resizing and filtering it.

• Finally pick the most "edgy" one, find its original and optionally convert it again:

  convert {$input} -thumbnail {$w}x{$h} -adaptive-sharpen 1.25x0.75 {$final_output}
  

I've found out that -adaptive-sharpen really helps with the end result because it sharpens the images only around those same edges. I tried different geometries and found out that 1.25x0.75 works best for me when I'm scaling down to a quarter of the original resolution.

I've done this in PHP and it takes about 25 seconds for a 12 minute movie to execute which is fine by me.

I hope this was helpful.

• http://www.imagemagick.org/Usage/transform/#edge
• http://www.imagemagick.org/Usage/blur/#related
• http://www.imagemagick.org/script/identify.php
• http://www.imagemagick.org/script/command-line-options.php#format_identify_

Comments

[Admin]

I tried using Face Detection as well. I theorized that if the identified face area overlaps then the image is blurry. I wasn't quite successful because it seems that faces are not easy to detect in video frames. The software requires almost "perfect" portrait photos.

[Admin]

I found this paper that might be relevant: research.microsoft.com/pubs/68802/… but it seems somewhat complex.

[Admin]

Does "blurry" for you mean that there are two overlapping images? Or do you have an example of a "blurry" thumbnail without any overlapping image? Note that there might be motion blur in the actual video itself—you can't get rid of that. If the person in the first image is actually moving, every still frame will exhibit a little motion blur.

[Admin]

The paper you found doesn't really have a lot to do with your problem. It describes a blurring metric, i.e. a metric to determine how blurry an image is (at the coding layer). But that won't help you a lot since you'd have to implement it yourself. Also the paper seems quite old and way the authors prove the validity of the metric doesn't look too convincing to me.

[Admin]

By "blurry", my top priority is overlapping images. That's the biggest problem. You pose an interesting question on whether there are blurry images that are not overlapping, but I haven't noticed that yet. I have found 2 other links: 1) stackoverflow.com/questions/5180327/… and 2) stackoverflow.com/questions/4470107/… The second link might be incorrect because my keyframes are from H.264 video.

[Admin]

So, to paraphrase your question: You'd like to find thumbnails based on the criteria that they're 1) keyframes and 2) show as little motion blur as possible. You're looking for any (objective) blurring metric then, which would calculate the amount of blur for each frame. To be honest, this is the kind of stuff that research institutions develop internally and try to market (as research innovation or sold to the industry). It's unlikely you'll find something readymade. The MSU Tool has a blurring metric though.

[Admin]

I've worked in the field of video quality for the last three years. I'll ask a colleague who has looked a little more into blurring metrics before and let you know if I'll find something useful. To be honest, if you're up for it you'd probably need to fire up MATLAB and do a little signal processing.

[Admin]

Thanks! I'm beginning to think I should generate maybe about 20 keyframes around the 3rd minute and then run image magik to compare 2 images (imagemagick.org/Usage/compare/#difference) Using the comparison, I should be able to (hopefully) eliminate overlapping images. I'm not sure this will work, but it will be intriguing if it does.

[cnfcnf]

I haven't tried this yet, but the logic seems sound and I am overall quite impressed with how thorough this has been. Thank you!

[matthewd]

Give it a try. I would be more than happy to see what your results are.