Adding frame to video with same FPS using FFMPEG concat reduces output FPS

Question

I am developing a piece of C# image-acquisition software in which I am compiling images into a video as I get them.

Currently I am doing this by creating a video from a set of images once the image count is that of the desired output FPS, for the sake of this question we will use 5 FPS. It will wait until it has 5 images, so it can render 1 second of footage for the base video, to which the new frames will be added. To generate this video I use this command:

-framerate 5 -start_number 1 -i "C:\path\to\image\Image_%05d.jpg" -c:v libx264 -pix_fmt yuv420p -r 5 "C:\path\to\output\output.mp4"

Probe output:

ffmpeg version N-86994-g92da230 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.1.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 74.100 / 55. 74.100
  libavcodec     57.102.100 / 57.102.100
  libavformat    57. 76.100 / 57. 76.100
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 99.100 /  6. 99.100
  libswscale      4.  7.102 /  4.  7.102
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
Input #0, image2, from 'B00375_EOL_%05d.jpg':
  Duration: 00:00:02.40, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 1200x1200 [SAR 144:144 DAR 1:1], 5 fps, 5 tbr, 5 tbn, 5 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[swscaler @ 0553cfc0] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 05491120] using SAR=1/1
[libx264 @ 05491120] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 05491120] profile High, level 4.0
[libx264 @ 05491120] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=5 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf57.76.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1200x1200 [SAR 1:1 DAR 1:1], q=-1--1, 5 fps, 10240 tbn, 5 tbc
    Metadata:
      encoder         : Lavc57.102.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=   12 fps=0.0 q=-1.0 Lsize=     187kB time=00:00:01.80 bitrate= 852.8kbits/s speed=2.76x
video:186kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.490686%
[libx264 @ 05491120] frame I:2     Avg QP:14.80  size: 23424
[libx264 @ 05491120] frame P:9     Avg QP:15.59  size: 14619
[libx264 @ 05491120] frame B:1     Avg QP:15.13  size: 11846
[libx264 @ 05491120] consecutive B-frames: 83.3% 16.7%  0.0%  0.0%
[libx264 @ 05491120] mb I  I16..4: 27.4% 71.2%  1.4%
[libx264 @ 05491120] mb P  I16..4: 15.6% 72.7%  0.2%  P16..4:  2.8%  0.8%  0.7%  0.0%  0.0%    skip: 7.2%
[libx264 @ 05491120] mb B  I16..4:  5.3% 31.8%  0.0%  B16..8: 13.6%  8.8%  0.7%  direct: 7.9%  skip:31.8%  L0:93.9% L1: 5.1% BI: 1.0%
[libx264 @ 05491120] 8x8 transform intra:80.1% inter:91.7%
[libx264 @ 05491120] coded y,uvDC,uvAC intra: 22.2% 25.1% 4.7% inter: 8.5% 14.4% 0.8%
[libx264 @ 05491120] i16 v,h,dc,p: 38% 31%  7% 24%
[libx264 @ 05491120] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 52% 21% 23%  1%  1%  0%  1%  1%  1%
[libx264 @ 05491120] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 30% 34% 14%  3%  3%  4%  3%  4%  3%
[libx264 @ 05491120] i8c dc,h,v,p: 65% 18% 14%  3%
[libx264 @ 05491120] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 05491120] ref P L0: 67.1%  9.6% 20.2%  3.1%
[libx264 @ 05491120] kb/s:634.23

From here I am then generating short single-frame videos to append onto the end of the video to add the new frame, this is done via this commmand:

-framerate 5 -loop 1 -i "C:\path\to\image\new_image.jpg" -c:v libx264 -pix_fmt yuv420p -t 0.2 -r 5 "C:\path\to\output\new_frame.mp4"

Probe output:

ffmpeg version N-86994-g92da230 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.1.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 74.100 / 55. 74.100
  libavcodec     57.102.100 / 57.102.100
  libavformat    57. 76.100 / 57. 76.100
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 99.100 /  6. 99.100
  libswscale      4.  7.102 /  4.  7.102
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
Input #0, image2, from 'B00375_PRC_00001.jpg':
  Duration: 00:00:00.20, start: 0.000000, bitrate: 880 kb/s
    Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 1200x1200 [SAR 144:144 DAR 1:1], 5 fps, 5 tbr, 5 tbn, 5 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[swscaler @ 03ba0fe0] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 03b0e2e0] using SAR=1/1
[libx264 @ 03b0e2e0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 03b0e2e0] profile High, level 4.0
[libx264 @ 03b0e2e0] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=5 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'new_frame.mp4':
  Metadata:
    encoder         : Lavf57.76.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1200x1200 [SAR 1:1 DAR 1:1], q=-1--1, 5 fps, 10240 tbn, 5 tbc
    Metadata:
      encoder         : Lavc57.102.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=    1 fps=0.0 q=23.0 Lsize=      15kB time=00:00:00.00 bitrate=1287020.4kbits/s speed=0.000332x
video:15kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 5.465249%
[libx264 @ 03b0e2e0] frame I:1     Avg QP:15.27  size: 14260
[libx264 @ 03b0e2e0] mb I  I16..4: 19.3% 79.8%  0.9%
[libx264 @ 03b0e2e0] 8x8 transform intra:79.8%
[libx264 @ 03b0e2e0] coded y,uvDC,uvAC intra: 19.1% 10.8% 0.1%
[libx264 @ 03b0e2e0] i16 v,h,dc,p: 35% 49%  6% 10%
[libx264 @ 03b0e2e0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 50% 22% 26%  0%  1%  0%  1%  0%  0%
[libx264 @ 03b0e2e0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 34% 21% 16%  4%  3%  6%  4%  5%  5%
[libx264 @ 03b0e2e0] i8c dc,h,v,p: 67% 18% 14%  1%
[libx264 @ 03b0e2e0] kb/s:570.40

Both of these output files are fine at this stage. The 'main' video generate lasts 1 second and shows 5 FPS in the file properties, so does the single frame, it lasts 0.2 seconds and shows 5 FPS in the properties. These files then get joined via the command:

-f concat -i concat.txt -auto_convert 1 -c copy joined_vid.mp4 Where concat.txt looks like:

file 'output.mp4'
file 'new_frame.mp4'

Probe output:

ffmpeg version N-86994-g92da230 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.1.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 74.100 / 55. 74.100
  libavcodec     57.102.100 / 57.102.100
  libavformat    57. 76.100 / 57. 76.100
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 99.100 /  6. 99.100
  libswscale      4.  7.102 /  4.  7.102
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 052d8700] Auto-inserting h264_mp4toannexb bitstream filter
Input #0, concat, from 'concat.txt':
  Duration: N/A, start: 0.000000, bitrate: 636 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1200x1200 [SAR 1:1 DAR 1:1], 636 kb/s, 5 fps, 5 tbr, 10240 tbn, 10 tbc
    Metadata:
      handler_name    : VideoHandler
Output #0, mp4, to 'joined_vid.mp4':
  Metadata:
    encoder         : Lavf57.76.100
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1200x1200 [SAR 1:1 DAR 1:1], q=2-31, 636 kb/s, 5 fps, 5 tbr, 10240 tbn, 10240 tbc
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mov,mp4,m4a,3gp,3g2,mj2 @ 052d8700] Auto-inserting h264_mp4toannexb bitstream filter
frame=   13 fps=0.0 q=-1.0 Lsize=     202kB time=00:00:02.40 bitrate= 690.0kbits/s speed= 135x
video:201kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.470814%

From here the newly created 'joined_vid.mp4's FPS drops down, usually to 4 FPS (but have seen it drop straight to 3 FPS) as shown in figure 1. As I incrementally add more frames and new images are acquired the FPS continues to drop until it gets to 1 FPS and then it stays on 1.

Figure 1

My question is how can I append my new image onto the end of the video without

• Dropping frames
• Having to re-render the whole video with each new image
• Dropping the output file's FPS

Any help is greatly appreciated as this currently has me stumped!

Solution

This is indeed due to the H264 PTS generation bug. My suggestion in the comment works for me. But so does the method at the end of the post.

Command for initial file:

ffmpeg -framerate 5 -start_number 1 -t 1 -i "f%d.png" -c:v libx264 -pix_fmt yuv420p -r 5 -bf 0 initial.mp4

Command for individual frames:

ffmpeg -framerate 5 -i "f6.png" -c:v libx264 -pix_fmt yuv420p -bf 0 -r 5 next1.mp4
...
ffmpeg -framerate 5 -i "f10.png" -c:v libx264 -pix_fmt yuv420p -bf 0 -r 5 next5.mp4

A single join:

list.txt:

file initial.mp4
file next1.mp4

and

ffmpeg -f concat -i list.txt -auto_convert 1 -c copy with6.mp4

with6.mp4:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'with6.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.2.100
  Duration: 00:00:01.20, start: 0.000000, bitrate: 129 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 124 kb/s, 5 fps, 5 tbr, 10240 tbn, 10 tbc (default)
    Metadata:
      handler_name    : VideoHandler

Join the rest at once:

file with6.mp4
file next2.mp4
file next3.mp4
file next4.mp4
file next5.mp4

and

ffmpeg -f concat -i list.txt -auto_convert 1 -c copy with6.mp4

with10.mp4:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'with10.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.2.100
  Duration: 00:00:02.00, start: 0.000000, bitrate: 168 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 165 kb/s, 5 fps, 5 tbr, 10240 tbn, 10 tbc (default)
    Metadata:
      handler_name    : VideoHandler

---

The other method is to keep your commands as-is but mux to MKV first, and then remux to MP4 i.e.

ffmpeg -f concat -i concat.txt -c copy new_output.mkv

ffmpeg -i new_output.mkv -c copy new_output.mp4

Comments

[Gyan]

Looks to be due to the broken H.264 PTS generation in ffmpeg. Workaround is to skip B frames, so add -bf 0 to your encoding commands.

[J. Scull]

@Mulvya sorry for the long reply, this has not fixed my issue and I am still seeing the same thing (frames going from 5 -> 1). Maybe the codecs are different on the appendix file to the time-lapse?

[Elisa Cha Cha]

Is there a bug report for this?

[Gyan]

For this specific issue? No, but #502 is related.

[J. Scull]

Thank you for this answer, it seems to work perfectly! What I misunderstood was the -bf 0, I only put it on when I was joining the videos as I assumed that's when I would need to skip the frames. I couldn't convert it to a different file type as all this processing has to occur very quickly. Cheers again :)