How frame rate is calculated or read from h264 bitstream sending through rtsp protocol by live555

Question

Problem:

In my case, bit stream coming from h264 encoder which is feed by live camera. Bitstream is sending by live555 library using RTSP protocol. RTSP packets are received by one of rtsp client like vlc player( the most important), totem player, mplayer. I need to know how to set lower framerate because my camera produces 9 fps but most of RTSP clients are expecting in default 25 fps and it cause that video stream is not played properly ( buffered, dropping late packets etc.).

Question:

I would like to know how frame rate is read from bit stream (or evaluated) by RTSP client (playback software). Where this informations are stored and how are sending to playback software?

More infos:

I am using testOnDemandRTSPServer example code from live555. I have wrote my implementation of DeviceSource class from live555 lib. I have set presentation time properly:

gettimeofday(&fPresentationTime, NULL);
fDurationInMicroseconds = 1100000;

Sequence Parameter Set NAL also contains proper timing infos:

timing_info_present_flag : 1 
       num_units_in_tick : 1001 
       time_scale : 9000 
       fixed_frame_rate_flag : 1

RTSP Header contains framerate:

a=framerate:9.0

Unfortunately it doesn't help. I have thought that video player will be calmly waiting for new frame and framerate will be correlated with frequency of receiving another RTP packets (another NALs).

How video players behave:

VLC player - video is choppy because vlc expects next frame earlier (25 fps instead of 9 fps). In results, vlc increases buffering time to e.g. 10s, during this time video is frozen.

Totem player - Timing informations from SPS Nal are parsed by totem. Framerate is showing properly but video is still choppy ( ~500ms delayed ). It seems like is waiting for more frames (25 per second).

Mplayer recognized wrong framerate value, shows 25 fps, but video is very smooth and good quality:

mplayer ffmpeg://rtsp://192.168.1.82:8554/testStream
.
.
.
VIDEO:  [H264]  320x240  0bpp  25.000 fps    0.0 kbps ( 0.0 kbyte/s)

Live555 RTSP client test application ./live555/testProgs/testRTSPClient rtsp://192.168.1.82:8554/testStream

Opening connection to 192.168.1.82, port 8554...
...remote connection opened
Sending request: DESCRIBE rtsp://192.168.1.82:8554/testStream RTSP/1.0
CSeq: 2
User-Agent: ./testRTSPClient (LIVE555 Streaming Media v2016.02.22)
Accept: application/sdp


Received 675 new bytes of response data.
Received a complete DESCRIBE response:
RTSP/1.0 200 OK
CSeq: 2
Date: Fri, Jan 01 1988 00:00:23 GMT
Content-Base: rtsp://192.168.1.82:8554/testStream/
Content-Type: application/sdp
Content-Length: 506

v=0
o=- 567993603275971 1 IN IP4 192.168.1.82
s=Session streamed by "testOnDemandRTSPServer"
i=testStream
t=0 0
a=tool:LIVE555 Streaming Media v2016.02.22
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "testOnDemandRTSPServer"
a=x-qt-text-inf:testStream
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42801E;sprop-parameter-sets=Z0KAHpWgUHxA,aM44gA==
a=framerate:9.0
a=control:track1

[URL:"rtsp://192.168.1.82:8554/testStream/"]: Got a SDP description:
v=0
o=- 567993603275971 1 IN IP4 192.168.1.82
s=Session streamed by "testOnDemandRTSPServer"
i=testStream
t=0 0
a=tool:LIVE555 Streaming Media v2016.02.22
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "testOnDemandRTSPServer"
a=x-qt-text-inf:testStream
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42801E;sprop-parameter-sets=Z0KAHpWgUHxA,aM44gA==
a=framerate:9.0
a=control:track1

[URL:"rtsp://192.168.1.82:8554/testStream/"]: Initiated the "video/H264" subsession (client ports 57660-57661)
Sending request: SETUP rtsp://192.168.1.82:8554/testStream/track1 RTSP/1.0
CSeq: 3
User-Agent: ./testRTSPClient (LIVE555 Streaming Media v2016.02.22)
Transport: RTP/AVP;unicast;client_port=57660-57661


Received 215 new bytes of response data.
Received a complete SETUP response:
RTSP/1.0 200 OK
CSeq: 3
Date: Fri, Jan 01 1988 00:00:23 GMT
Transport: RTP/AVP;unicast;destination=192.168.1.109;source=192.168.1.82;client_port=57660-57661;server_port=6970-6971
Session: 06B1DFA4;timeout=65


[URL:"rtsp://192.168.1.82:8554/testStream/"]: Set up the "video/H264" subsession (client ports 57660-57661)
[URL:"rtsp://192.168.1.82:8554/testStream/"]: Created a data sink for the "video/H264" subsession
Sending request: PLAY rtsp://192.168.1.82:8554/testStream/ RTSP/1.0
CSeq: 4
User-Agent: ./testRTSPClient (LIVE555 Streaming Media v2016.02.22)
Session: 06B1DFA4
Range: npt=0.000-


Received 96 new bytes of response data.
Received 94 new bytes of response data.
Received a complete PLAY response:
RTSP/1.0 200 OK
CSeq: 4
Date: Fri, Jan 01 1988 00:00:23 GMT
Range: npt=0.000-
Session: 06B1DFA4
RTP-Info: url=rtsp://192.168.1.82:8554/testStream/track1;seq=52833;rtptime=3161001994


[URL:"rtsp://192.168.1.82:8554/testStream/"]: Started playing session...
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 10874 bytes.   Presentation time: 567993623.442288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12054 bytes.   Presentation time: 567993623.482288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12227 bytes.   Presentation time: 567993623.522288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12281 bytes.   Presentation time: 567993623.562288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12384 bytes.   Presentation time: 567993623.602288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12129 bytes.   Presentation time: 567993623.642288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12141 bytes.   Presentation time: 567993623.682288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12287 bytes.   Presentation time: 567993623.722288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 16415 bytes.   Presentation time: 567993623.762288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12360 bytes.   Presentation time: 567993623.802288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264:      Received 12167 bytes.   Presentation time: 567993623.842288

I parsed my NALs by tool h264_analyze and below is output ( only a few first NALs):

!! Found NAL at offset 4 (0x0004), size 18 (0x0012) 
XX 00 00 00 01 67 42 80 1E 95 A0 50 7C 84 00 00 0F 

==================== NAL ====================
 forbidden_zero_bit : 0 
 nal_ref_idc : 3 
 nal_unit_type : 7 ( Sequence parameter set ) 
======= SPS =======
 profile_idc : 66 
 constraint_set0_flag : 1 
 constraint_set1_flag : 0 
 constraint_set2_flag : 0 
 constraint_set3_flag : 0 
 constraint_set4_flag : 0 
 constraint_set5_flag : 0 
 reserved_zero_2bits : 0 
 level_idc : 30 
 seq_parameter_set_id : 0 
 chroma_format_idc : 1 
 residual_colour_transform_flag : 0 
 bit_depth_luma_minus8 : 0 
 bit_depth_chroma_minus8 : 0 
 qpprime_y_zero_transform_bypass_flag : 0 
 seq_scaling_matrix_present_flag : 0 
 log2_max_frame_num_minus4 : 4 
 pic_order_cnt_type : 2 
   log2_max_pic_order_cnt_lsb_minus4 : 0 
   delta_pic_order_always_zero_flag : 0 
   offset_for_non_ref_pic : 0 
   offset_for_top_to_bottom_field : 0 
   num_ref_frames_in_pic_order_cnt_cycle : 0 
 num_ref_frames : 1 
 gaps_in_frame_num_value_allowed_flag : 0 
 pic_width_in_mbs_minus1 : 19 
 pic_height_in_map_units_minus1 : 14 
 frame_mbs_only_flag : 1 
 mb_adaptive_frame_field_flag : 0 
 direct_8x8_inference_flag : 0 
 frame_cropping_flag : 0 
   frame_crop_left_offset : 0 
   frame_crop_right_offset : 0 
   frame_crop_top_offset : 0 
   frame_crop_bottom_offset : 0 
 vui_parameters_present_flag : 1 
=== VUI ===
 aspect_ratio_info_present_flag : 0 
   aspect_ratio_idc : 0 
     sar_width : 0 
     sar_height : 0 
 overscan_info_present_flag : 0 
   overscan_appropriate_flag : 0 
 video_signal_type_present_flag : 0 
   video_format : 0 
   video_full_range_flag : 0 
   colour_description_present_flag : 0 
     colour_primaries : 0 
   transfer_characteristics : 0 
   matrix_coefficients : 0 
 chroma_loc_info_present_flag : 0 
   chroma_sample_loc_type_top_field : 0 
   chroma_sample_loc_type_bottom_field : 0 
 timing_info_present_flag : 1 
   num_units_in_tick : 1001 
   time_scale : 9000 
   fixed_frame_rate_flag : 1
 nal_hrd_parameters_present_flag : 0 
 vcl_hrd_parameters_present_flag : 0 
   low_delay_hrd_flag : 0 
 pic_struct_present_flag : 0 
 bitstream_restriction_flag : 0 
   motion_vectors_over_pic_boundaries_flag : 0 
   max_bytes_per_pic_denom : 0 
   max_bits_per_mb_denom : 0 
   log2_max_mv_length_horizontal : 0 
   log2_max_mv_length_vertical : 0 
   num_reorder_frames : 0 
   max_dec_frame_buffering : 0 
=== HRD ===
 cpb_cnt_minus1 : 0 
 bit_rate_scale : 0 
 cpb_size_scale : 0 
   bit_rate_value_minus1[0] : 0 
   cpb_size_value_minus1[0] : 0 
   cbr_flag[0] : 0 
 initial_cpb_removal_delay_length_minus1 : 0 
 cpb_removal_delay_length_minus1 : 0 
 dpb_output_delay_length_minus1 : 0 
 time_offset_length : 0 
!! Found NAL at offset 26 (0x001A), size 4 (0x0004) 
XX 00 00 00 01 68 CE 38 80 
==================== NAL ====================
 forbidden_zero_bit : 0 
 nal_ref_idc : 3 
 nal_unit_type : 8 ( Picture parameter set ) 
======= PPS =======
 pic_parameter_set_id : 0 
 seq_parameter_set_id : 0 
 entropy_coding_mode_flag : 0 
 pic_order_present_flag : 0 
 num_slice_groups_minus1 : 0 
 slice_group_map_type : 0 
 num_ref_idx_l0_active_minus1 : 0 
 num_ref_idx_l1_active_minus1 : 0 
 weighted_pred_flag : 0 
 weighted_bipred_idc : 0 
 pic_init_qp_minus26 : 0 
 pic_init_qs_minus26 : 0 
 chroma_qp_index_offset : 0 
 deblocking_filter_control_present_flag : 0 
 constrained_intra_pred_flag : 0 
 redundant_pic_cnt_present_flag : 0 
 transform_8x8_mode_flag : 0 
 pic_scaling_matrix_present_flag : 0 
 second_chroma_qp_index_offset : 0 
!! Found NAL at offset 33 (0x0021), size 339 (0x0153) 
XX 80 00 00 01 65 B8 04 04 3F FF F8 7A 28 03 EF BE 

==================== NAL ====================
 forbidden_zero_bit : 0 
 nal_ref_idc : 3 
 nal_unit_type : 5 ( Coded slice of an IDR picture ) 
======= Slice Header =======
 first_mb_in_slice : 0 
 slice_type : 2 ( I slice ) 
 pic_parameter_set_id : 0 
 frame_num : 0 
 field_pic_flag : 0 
 bottom_field_flag : 0 
 idr_pic_id : 0 
 pic_order_cnt_lsb : 0 
 delta_pic_order_cnt_bottom : 0 
 redundant_pic_cnt : 0 
 direct_spatial_mv_pred_flag : 0 
 num_ref_idx_active_override_flag : 0 
 num_ref_idx_l0_active_minus1 : 0 
 num_ref_idx_l1_active_minus1 : 0 
 cabac_init_idc : 0 
 slice_qp_delta : -16 
 sp_for_switch_flag : 0 
 slice_qs_delta : 0 
 disable_deblocking_filter_idc : 0 
 slice_alpha_c0_offset_div2 : 0 
 slice_beta_offset_div2 : 0 
 slice_group_change_cycle : 0 
=== Prediction Weight Table ===
 luma_log2_weight_denom : 0 
 chroma_log2_weight_denom : 0 
=== Ref Pic List Reordering ===
 ref_pic_list_reordering_flag_l0 : 0 
 ref_pic_list_reordering_flag_l1 : 0 
=== Decoded Ref Pic Marking ===
 no_output_of_prior_pics_flag : 0 
 long_term_reference_flag : 0 
 adaptive_ref_pic_marking_mode_flag : 0 
!! Found NAL at offset 376 (0x0178), size 1897 (0x0769) 
XX 00 00 00 01 41 E0 20 3F 36 5F FF 87 A2 80 03 7E 

==================== NAL ====================
 forbidden_zero_bit : 0 
 nal_ref_idc : 2 
 nal_unit_type : 1 ( Coded slice of a non-IDR picture ) 
======= Slice Header =======
 first_mb_in_slice : 0 
 slice_type : 0 ( P slice ) 
 pic_parameter_set_id : 0 
 frame_num : 1 
 field_pic_flag : 0 
 bottom_field_flag : 0 
 idr_pic_id : 0 
 pic_order_cnt_lsb : 0 
 delta_pic_order_cnt_bottom : 0 
 redundant_pic_cnt : 0 
 direct_spatial_mv_pred_flag : 0 
 num_ref_idx_active_override_flag : 0 
 num_ref_idx_l0_active_minus1 : 0 
 num_ref_idx_l1_active_minus1 : 0 
 cabac_init_idc : 0 
 slice_qp_delta : -15 
 sp_for_switch_flag : 0 
 slice_qs_delta : 0 
 disable_deblocking_filter_idc : 0 
 slice_alpha_c0_offset_div2 : 0 
 slice_beta_offset_div2 : 0 
 slice_group_change_cycle : 0 
=== Prediction Weight Table ===
 luma_log2_weight_denom : 0 
 chroma_log2_weight_denom : 0 
=== Ref Pic List Reordering ===
 ref_pic_list_reordering_flag_l0 : 0 
 ref_pic_list_reordering_flag_l1 : 0 
=== Decoded Ref Pic Marking ===
 no_output_of_prior_pics_flag : 0 
 long_term_reference_flag : 0 
 adaptive_ref_pic_marking_mode_flag : 0 
!! Found NAL at offset 2277 (0x08E5), size 2989 (0x0BAD) 
XX 00 00 00 01 41 E0 40 3F 37 0F 0C 3E D8 A0 00 4E 

==================== NAL ====================
 forbidden_zero_bit : 0 
 nal_ref_idc : 2 
 nal_unit_type : 1 ( Coded slice of a non-IDR picture ) 
======= Slice Header =======
 first_mb_in_slice : 0 
 slice_type : 0 ( P slice ) 
 pic_parameter_set_id : 0 
 frame_num : 2 
 field_pic_flag : 0 
 bottom_field_flag : 0 
 idr_pic_id : 0 
 pic_order_cnt_lsb : 0 
 delta_pic_order_cnt_bottom : 0 
 redundant_pic_cnt : 0 
 direct_spatial_mv_pred_flag : 0 
 num_ref_idx_active_override_flag : 0 
 num_ref_idx_l0_active_minus1 : 0 
 num_ref_idx_l1_active_minus1 : 0 
 cabac_init_idc : 0 
 slice_qp_delta : -15 
 sp_for_switch_flag : 0 
 slice_qs_delta : 0 
 disable_deblocking_filter_idc : 0 
 slice_alpha_c0_offset_div2 : 0 
 slice_beta_offset_div2 : 0 
 slice_group_change_cycle : 0 
=== Prediction Weight Table ===
 luma_log2_weight_denom : 0 
 chroma_log2_weight_denom : 0 
=== Ref Pic List Reordering ===
 ref_pic_list_reordering_flag_l0 : 0 
 ref_pic_list_reordering_flag_l1 : 0 
=== Decoded Ref Pic Marking ===
 no_output_of_prior_pics_flag : 0 
 long_term_reference_flag : 0 
 adaptive_ref_pic_marking_mode_flag : 0 
!! Found NAL at offset 5270 (0x1496), size 4074 (0x0FEA) 
XX 00 00 00 01 41 E0 60 10 CD B7 C1 BF B8 36 00 6A 

Solution 1

I would like to know how frame rate is read from bit stream (or evaluated) by RTSP client (playback software). Where this informations are stored and how are sending to playback software?

The client does not need to know the framerate to playback the received video. Playback is according to presentation time which is calculated from RTP and NTP timestamps.

Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 10874 bytes. Presentation time: 567993623.442288

Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12054 bytes. Presentation time: 567993623.482288

It does not look like you are setting your presentation time correctly. That's likely where your 25fps per second are coming in. If they were set correctly testRTSPClient would print 9 samples per second with an approximate difference of 1/9 second between them. Your differences are about 40ms giving you the 25fps.

a=framerate:9.0

I've developed RTSP servers using various frame rate content, and I've never needed to set that SDP attribute. I'm not sure what client software actually make use of it.

VLC player - video is choppy because vlc expects next frame earlier (25 fps instead of 9 fps). In results, vlc increases buffering time to e.g. 10s, during this time video is frozen.

In my experience over the last few years VLC is bad at playing back live content. I've tested various RTSP servers and VLC often runs into problems, even when using bigger network jitter buffers.

A few other things you might want to look at:

fDurationInMicroseconds = 1100000;

As stated in the comment, fDurationInMicroseconds should be set to 0 for live sources.

b=AS:500

The bitrate seems to be set incorrectly: your stream is definitely not 500kbps.

Solution 2

Problem solved. I was using wrong class for streaming. Described here: http://lists.live555.com/pipermail/live-devel/2016-April/020013.html

According to Ross hint I replaced H264VideoStreamFramer to H264VideoStreamDiscreteFramer and I removed start code from NAL units. Everything works perfectly now.

Comments

[Ralf]

AFAIR fDurationInMicroseconds should be set to 0 for live sources. Not sure if that will affect the outcome though.

[Pawel Domagalski]

You have right with timestamps. RTSP client receives RTP packets timestamped with 40 ms. I have checked that presentationTime is set correctly by me on server site but live555 library is processing this values incorrectly. Probably I have different system timer frequency on my development board. Should be 90Khz. Do you know how live555 handles timestamps ? Thanks for help