Screen capture using ffmpeg -f avfoundation and libvpx too many problems

Question

I'm trying to capture screen and audio on Mac OSX 10.12.2. I prefer webm format because it can be easily fragmented and streamed. Also cpu usage is good. But at this stage where nothing seems to work for me. I'll accept recording in mp4/h265/aac.

After trying for 10 long hours and hitting my head to the wall many times, these are the results I'm getting:

vp8/vorbis: no sound and video plays too fast

ffmpeg  -y -f avfoundation -capture_cursor 1 -framerate 25 -r 25 -i 1:0 -vsync 2 -c:v libvpx -c:a libvorbis -b:v 1M -b:a 128K capture.webm

output:

ffmpeg version 3.2.2 Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-opencl --disable-lzma --enable-nonfree --enable-vda
  libavutil      55. 34.100 / 55. 34.100
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.100 / 57. 56.100
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
[avfoundation @ 0x7fc5c2000000] Selected pixel format (yuv420p) is not supported by the input device.
[avfoundation @ 0x7fc5c2000000] Supported pixel formats:
[avfoundation @ 0x7fc5c2000000]   uyvy422
[avfoundation @ 0x7fc5c2000000]   yuyv422
[avfoundation @ 0x7fc5c2000000]   nv12
[avfoundation @ 0x7fc5c2000000]   0rgb
[avfoundation @ 0x7fc5c2000000]   bgr0
[avfoundation @ 0x7fc5c2000000] Overriding selected pixel format to use uyvy422 instead.
[avfoundation @ 0x7fc5c2000000] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, avfoundation, from '1:0':
  Duration: N/A, start: 25068.515833, bitrate: 2822 kb/s
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 2560x1600, 1000k tbr, 1000k tbn, 1000k tbc
    Stream #0:1: Audio: pcm_f32le, 44100 Hz, stereo, flt, 2822 kb/s
[libvpx @ 0x7fc5c2842e00] v1.6.0
Output #0, webm, to 'capture.webm':
  Metadata:
    encoder         : Lavf57.56.100
    Stream #0:0: Video: vp8 (libvpx), yuv420p, 2560x1600, q=-1--1, 1000 kb/s, 25 fps, 1k tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.64.101 libvpx
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: vorbis (libvorbis), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc57.64.101 libvorbis
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> vp8 (libvpx))
  Stream #0:1 -> #0:1 (pcm_f32le (native) -> vorbis (libvorbis))
Press [q] to stop, [?] for help
frame=   10 fps=0.8 q=0.0 Lsize=     240kB time=00:00:00.36 bitrate=5456.9kbits/s speed=0.0284x
video:236kB audio:0kB subtitle:0kB other streams:0kB global headers:4kB muxing overhead: 2.017218%

vp9/opus: no sound and video plays too fast, sometimes ffmpeg hangs, cannot quit

ffmpeg  -y -f avfoundation -capture_cursor 1 -framerate 30 -r 30 -i 1:0 -c:v libvpx-vp9 -c:a libopus -crf 10 -b:v 1M -b:a 128K capture.webm

No output, hard exit using ctrl+c.

mp4: poor audio quality, it stutters, also too much cpu

ffmpeg -y -f avfoundation -capture_cursor 1 -video_size 720x480 -framerate 25 -i 1:0 -vsync 2 -preset slow -crf 22 -c:v libx264 -c:a aac -b:v 1M -b:a 128K capture.mp4

output:

ffmpeg version N-82977-g8fd3e02-tessus Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      55. 43.100 / 55. 43.100
  libavcodec     57. 70.100 / 57. 70.100
  libavformat    57. 61.100 / 57. 61.100
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 68.100 /  6. 68.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
[avfoundation @ 0x7ff6c9800000] Selected pixel format (yuv420p) is not supported by the input device.
[avfoundation @ 0x7ff6c9800000] Supported pixel formats:
[avfoundation @ 0x7ff6c9800000]   uyvy422
[avfoundation @ 0x7ff6c9800000]   yuyv422
[avfoundation @ 0x7ff6c9800000]   nv12
[avfoundation @ 0x7ff6c9800000]   0rgb
[avfoundation @ 0x7ff6c9800000]   bgr0
[avfoundation @ 0x7ff6c9800000] Overriding selected pixel format to use uyvy422 instead.
[avfoundation @ 0x7ff6c9800000] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, avfoundation, from '1:0':
  Duration: N/A, start: 27439.487000, bitrate: 2822 kb/s
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 2560x1600, 1000k tbr, 1000k tbn, 1000k tbc
    Stream #0:1: Audio: pcm_f32le, 44100 Hz, stereo, flt, 2822 kb/s
No pixel format specified, yuv422p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x7ff6c9839600] MB rate (16000000000) > level limit (2073600)
[libx264 @ 0x7ff6c9839600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x7ff6c9839600] profile High 4:2:2, level 5.2, 4:2:2 8-bit
[libx264 @ 0x7ff6c9839600] 264 - core 148 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=5 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=8 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=3 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=50 rc=crf mbtree=1 crf=22.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'capture.mp4':
  Metadata:
    encoder         : Lavf57.61.100
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv422p, 2560x1600, q=-1--1, 1000 kb/s, 1000k fps, 1000k tbn, 1000k tbc
    Metadata:
      encoder         : Lavc57.70.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: aac (LC) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc57.70.100 aac
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (pcm_f32le (native) -> aac (native))
Press [q] to stop, [?] for help
frame=  158 fps= 13 q=-1.0 Lsize=    1244kB time=00:00:09.66 bitrate=1054.7kbits/s speed=0.796x
video:1200kB audio:39kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.466263%
[libx264 @ 0x7ff6c9839600] frame I:1     Avg QP:18.97  size:338028
[libx264 @ 0x7ff6c9839600] frame P:49    Avg QP:20.98  size: 15709
[libx264 @ 0x7ff6c9839600] frame B:108   Avg QP:25.01  size:  1112
[libx264 @ 0x7ff6c9839600] consecutive B-frames:  5.1%  8.9%  7.6% 78.5%
[libx264 @ 0x7ff6c9839600] mb I  I16..4: 21.7% 41.3% 37.0%
[libx264 @ 0x7ff6c9839600] mb P  I16..4:  0.4%  0.3%  0.6%  P16..4:  4.8%  0.6%  0.7%  0.0%  0.0%    skip:92.6%
[libx264 @ 0x7ff6c9839600] mb B  I16..4:  0.1%  0.1%  0.0%  B16..8:  2.9%  0.2%  0.1%  direct: 0.0%  skip:96.7%  L0:49.7% L1:49.5% BI: 0.8%
[libx264 @ 0x7ff6c9839600] 8x8 transform intra:35.8% inter:23.9%
[libx264 @ 0x7ff6c9839600] direct mvs  spatial:96.3% temporal:3.7%
[libx264 @ 0x7ff6c9839600] coded y,uvDC,uvAC intra: 30.5% 38.8% 34.6% inter: 0.3% 1.3% 0.4%
[libx264 @ 0x7ff6c9839600] i16 v,h,dc,p: 53% 43%  3%  1%
[libx264 @ 0x7ff6c9839600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 57%  5% 33%  1%  1%  1%  1%  1%  2%
[libx264 @ 0x7ff6c9839600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 35% 19% 14%  4%  5%  5%  4%  6%  7%
[libx264 @ 0x7ff6c9839600] i8c dc,h,v,p: 65% 14% 17%  3%
[libx264 @ 0x7ff6c9839600] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7ff6c9839600] ref P L0: 70.4%  2.2% 17.0%  6.3%  3.5%  0.7%
[libx264 @ 0x7ff6c9839600] ref B L0: 65.0% 22.6%  7.2%  5.1%
[libx264 @ 0x7ff6c9839600] ref B L1: 90.2%  9.8%
[libx264 @ 0x7ff6c9839600] kb/s:1014.78
[aac @ 0x7ff6ca809e00] Qavg: 581.188
Exiting normally, received signal 15.

I would like to know if I'm missing out any important option that would make the video playable or if this is a bug in ffmpeg.

Solution

Your problem seems to be compute bound, i.e running out of CPU cycles for encoding the requested formats.

A somewhat viable solution would be to capture the stream on the host Mac OSX machine and then pipe it out to a much faster machine for encoding via netcat.

So, on the capture machine, you could run:

ffmpeg -y -f avfoundation -capture_cursor 1 -video_size 720x480 -framerate 25 -i 1:0 -vsync 2 -c:v copy -c:a copy -f mpegts - | nc -l -p 9000

That process would pipe out an MPEG TS stream to port 9000 via netcat.

And now, on the encoding machine,receive the mpeg ts video stream via netcat,feed it to ffmpeg via a unix pipe, and then you can adapt the following examples:

(a). For VAAPI, on supported hardware, transcode to H.264,scaled to the requested resolution:

nc <host-ip> 9000 | ffmpeg -i pipe:0 -vaapi_device /dev/dri/renderD129 -vf 'format=nv12,hwupload,scale_vaapi=w=720:h=480' -c:v h264_vaapi -qp:v 19 -b:v 1M -bf:v 4 -threads 4 -aspect 16:9 -c:a aac -b:a 128k -f mp4 -y "capture.mp4"

For any other encoders available on the more powerful workstation, tune them as necessary.

Regards,

Brainiarc7