FFMpeg: Add a circular mask over a video

Question

I have a video that I want to mask within a circle in the center of the video. I've found I can do a square mask, but how could I do a circle (without referencing an image to mask over it)?

Solution

Assuming you are using the inverted mask, use

ffmpeg -y -i video.mp4 -loop 1 -t 1 -i mask.png
  -filter_complex
      "color=black:d=1[c];[c][0]scale2ref[cs][v];[cs]setsar=1[ct];
       [1:v]alphaextract,negate[m];[m][ct]scale2ref[ms][ol];[ms]setsar=1[alf];
       [ol][alf]alphamerge[fin];
       [v][fin]overlay,scale‌​=640:1136:force_orig‌​inal_aspect_ratio=de‌​crease[fv];
       [fv]pad=6‌​40:1136:(ow-iw)/2:(o‌​h-ih)/2:#000000@1[v]‌​"
-map "[v]" -map 0:a output.mp4

First, a black overlay is created. It's resized to the video size. Then the mask is resized to the overlay size. Then its alpha channel is extracted and negated; merged with the color overlay. The result is overlaid on the video, the result resized and padded with black as needed.

If the mask isn't inverted, remove the negate.

Comments

[Gyan]

Just to make sure our lines aren't crossed - can you show a sample frame of the output mocked up elsewhere? In the ZIP file, the video is already within a circle, and the alpha channel in your mask PNG is inverted i.e. the black circle in the RGB channel of the PNG is opaque not transparent. After applying the alpha to the color output, that portion is shown.

[Gyan]

Also, your mask and video aren't the same size.

[Gyan]

If you post a sample image of the required output, I can show the command with mask resizing.

[Gyan]

In the latest ZIP, the source video is already circular and the result has no white padding. That suggests, all you need is scale and pad. No mask involved. Is that right?

[Gyan]

Command changed and tested.

[rraallvv]

I'm getting this error [AVFilterGraph @ 0x7fbd57d033c0] No such filter: 'scale‌​' Error initializing complex filters. Invalid argument

[Gyan]

Looks like your binary wasn't compiled with scale. Get one from ffmpeg.zeranoe.com/builds or johnvansickle.com/ffmpeg