User Generated Chroma Key Composite Videos with FFMPEG / AVCONV

Question

I want to create a system where a user can upload two videos to a website, one of which has blue screen in the background, and the website uses AVCONV (or similar) to create a composite video of the two using the Chroma Key technique (aka blue screening).

Can AVCONV be used in this way?

According to this post on SO a static image can be Chrome Keyed into the background. But can the same be done with a video?

I would also like to know if the two videos would need to be of the same length and format before they could be merged? Are there any other technical considerations I should be looking into to be able to offer a service like this?

Thanks in advance.

Solution 1

Note: This answer was posted before FFmpeg gained the capability to do chroma keying. See other answers below.

Neither AVconv nor FFmpeg can currently do this - it's the one major video processing task that they cannot do.

These tools can overlay an image or video with a transparent background over another video or image - which is what that SO question you linked to wanted. However, the video with the transparent background needs to be in a video format that supports transparency (most of them don't) - I'd use qtrle (Apple's Animation codec). But they cannot do the actual chroma-keying (turning a certain colour transparent).

You could probably use a workaround involving avconv and ImageMagick, but it would be rather CPU-intensive, and I don't know how good your results would be. You would need to:

1. Turn the video you want to chroma-key into individual frames with avconv
2. Use ImageMagick's convert (or mogrify) tool to do the chroma-keying on each individual frame
3. Stitch the frames back together with avconv & place it over the background video

You may be able to simplify the process using pipes.

To extract the frames (produces files that look like 00001.png, 00002.png, etc):

avconv -i input.mp4 %05d.png

To stitch them back together & put on top of your background video after you've run ImageMagick over them:

avconv -f image2 -r 25 -i %05d.png -i background.mp4 \
-filter_complex '[1:v][0:v]overlay[out]' \
-map [out] -c:v libx264 -crf 23 -preset veryfast output.mp4

-r 25 gives you a frame rate of 25; if you want another frame rate, just use a different number. Make sure you use the same frame rate as the original input video. If you want to take the audio from your original input, use

avconv -f image2 -r 25 -i %05d.png -i background.mp4 -i input.mp4 \
-filter_complex '[1:v][0:v]overlay[out]' \
-map [out] -map 2:a -c:v libx264 -crf 23 -preset veryfast -c:a copy output.mp4

Note that some versions of AVconv (like the version in the Ubuntu repos) don't have support for filtergraphs. If that's the case for you, upgrade to a newer version of avconv or ffmpeg.

As for stage 2... I'm actually not sure how to do it, but I know that ImageMagick can do chroma-keying. Here are a few links you may find useful:

http://www.imagemagick.org/Usage/photos/#chroma_key
http://www.imagemagick.org/discourse-server/viewtopic.php?t=14394
http://tech.natemurray.com/2007/12/convert-white-to-transparent.html

Due to differences in lighting etc between the samples people will give you, you'll have to use something to detect the exact shades of blue to get rid of. You'll probably have to leave that up to your users, perhaps using ffmpeg to create a thumbnail, and then using an eyedropper to choose the background colour.

Solution 2

You can do this using MLT

melt https://archive.org/download/IMB_SF_R35_C8/IMB_SF_R35_C8_512kb.mp4 in=400 \
    -track https://archive.org/download/IMB_SF_R19_C9/IMB_SF_R19_C9_512kb.mp4 in=450 \
    -filter chroma key=0x00ff0000 variance=0.45 \
    -transition composite

This plays a background video and a green-screen clip of a dog as a second track composited on top. It uses a chroma key (set to green with some variance) to apply the alpha to the green areas allowing the composited background to show through. The in=XXX are just to truncate off some leading garbage on the video clips being used.

Solution 3

chroma key is available in ffmpeg from version 3.0 ,before it you can use colorkey option

ffmpeg -f mpeg -i input.mp4 -i video.mp4 -shortest -filter_complex "[1:v]chromakey=0x70de77:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" output.mp4

for colorkey in ffmpeg v2.8 or above

 ffmpeg -f mpeg -i input.mp4 -i video.mp4 -shortest -filter_complex "[1:v]colorkey=0x70de77:0.1:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" output.mp

Both of these are working for me.

Comments

[Jimmery]

So, technically cant be done, but you have found an excellent work around. Thank you very much!

[evilsoup]

Glad to be of help :) ...if you do use this method, could you please post how you did it here, as an answer to help other people? I'd also be personally interested.

[Elisa Cha Cha]

The frei0r filters in ffmpeg may be able to do this, but I'm unsure ffmpeg can support using frei0r filters that use two inputs. Talking about frei0r, I'm also unsure if ffmpeg can support the "mixer2" and "mixer3" frei0r filters.

[evilsoup]

@LordNeckbeard that's a really good point, I'm currently compiling a new version of ffmpeg with --enable-libfrei0r, will test out the bluescreen0r filter later today/tomorrow.

[Elisa Cha Cha]

I'll be interested in your results.

[Thirumalai murugan]

@evilsoup I am looking for the overlay on live video, at a time I want to see a three live video in a single stream is it possible, if so could you help by giving the code?

[evilsoup]

@Thirumalaimurugan I would recommend asking that as a separate question (maybe linking to this one if you think it's relevant), that way you can go into more detail and the other people will get a look at the problem

[Thirumalai murugan]

@evilsoup superuser.com/questions/581386/avconv-overlay-for-live-video I have posted the new question please have a look and revert me back thank you

[Michael]

what's the difference between colorkey and chromakey? the latter seems to splash noisy green splotches all over the video whereas the former does not (for the same settings)