FFMPEG: chroma key / greenscreen filter for images / video

Question

I need for a video to use chroma key filter / greenscreen filter and overlay it over another video and output it as new video.

Are there any existing libraries, scripts, filters or solutions for this purpose?

Solution 1

The answer (now) is yes, there is a filter for generating chroma-keys and overlaying them. The filter name is "color key". There are examples on the site, here's the command:

ffmpeg -i <base-video> -i <overlay-video> -filter_complex '[1:v]colorkey=0x<color>:<similarity>:<blend>[ckout];[0:v][ckout]overlay[out]' -map '[out]' <output-file>

where <color> is the rgb color to match in hex (ex: 0x000000 for black), <similarity> is a tolerance on the color-match (ex: 0.3), and <blend> (ex: 0.2) controls whether the opacity is on-off or how gradual it is. (See the documentation for more).

Solution 2

Minimal runnable example with test data

The answer at https://stackoverflow.com/a/32291842/895245 was correct, here's just a minimal concrete example of that.

Download input media:

wget https://github.com/cirosantilli/media/raw/master/Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg
wget https://github.com/cirosantilli/media/raw/master/opengl-rotating-triangle.mp4

Make the image size match the video size of 1024x1024. The video size can be determined with ffprobe:

convert Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg -resize 1024x1024! background.jpg

Do the actual conversion:

ffmpeg -i background.jpg -i opengl-rotating-triangle.mp4 \
  -filter_complex '[1:v]colorkey=0x000000:0.1:[ckout];[0:v][ckout]overlay[out]' \
  -map '[out]' out.mp4

Convert to gif just for previews on this answer:

ffmpeg -i out.mp4 -r 5 -vf "scale=300:-1,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" out.gif

So in my example, I had a black background, which is getting converted into a fixed image.

Outcome preview (horrendous FPS to fit GIF in 2MB for upload here):

Actual video output: https://www.youtube.com/watch?v=3aY6x7u86QQ

Original input files for reference:

opengl-rotating-triangle.mp4

Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg

It also just works with a video background.

wget https://upload.wikimedia.org/wikipedia/commons/f/f9/STS-132_Liftoff_Space_Shuttle_Atlantis.ogv
ffmpeg -i STS-132_Liftoff_Space_Shuttle_Atlantis.ogv -i opengl-rotating-triangle.mp4 -filter_complex '[1:v]colorkey=0x000000:0.1:[ckout];[0:v]trim=start=0:end=8[cut0];[cut0][ckout]overlay[out]' -map '[out]' out.mp4

Here I've added another the parameter shortest=1 to the overlay filter:

overlay=shortest=1

to make the overlay stop as soon as the triangle video stops, otherwise it defaults to stopping when the much longer Atlantis video ends.

You might also want to use the trim filter instead to select an arbitrary segment.

The Atlantis video is 1920x1080, and the final output had that size as well. Preview:

Actual video: https://www.youtube.com/watch?v=HI8XniA2Bk8

Tested on Ubuntu 20.10, FFmpeg 4.3.1.

Solution 3

Not specifically that I know of

Opencv contains all the functions you need to read video, convert to RGB, split the color planes, replace pixels base don color, merge frames and write video.

It's good to research with but it's not going to create a plugin directshow filter to do this automatically

Comments

[Daniel Ruf]

might be much easier than Opencv?

[Tahir Akhtar]

I can't comment on this, may be you can ask this as a separate question so someone with experience of both frameworks can reply

[Daniel Ruf]

Great thanks. So we have now a direct solution using ffmpeg. Since which version of mmpeg is this available?

[jladan]

It was added to the master branch June 13, but hasn't made it into an actual release yet (current release is 2.7.2). One would need to build ffmpeg from source.

[matteo]

I downloaded the master branch of ffmpeg from github and compiled it, yet it still tells me "Unrecognized option '-filter_complex'"

[matteo]

I see, it's '-filter_complex', not '--filter_complex' (only one dash)

[matteo]

How would you change this to use an image as background? I tried by simply replacing the background video with an image file, but it results in a one-frame output, and adding -stream_loop -1 doesn't change a thing.

[jladan]

According to all the documentation (on both overlay and colorkey), it should work when one of the inputs is an image. You might try changing the order of the inputs (and swapping the 1 and 0 in the filter complex). Another option is to create a video out of the image, and use that video as the background.

[Ryk Waters]

Does anyone know if this has made it in to a release yet?

[Gyan]

Yes. Use the current release - 3.1.x

[Aakash]

This helped, but instead of colorkey the new versions use chromakey filters.

[Michael]

Why is this keying out parts of my face (not green at all!) while parts of the green screen with different brightness as still plainly visible?