ffmpeg.c what are pts and dts ? what does this code block do in ffmpeg.c?

Question

• In simple terms what are pts and dts values?
• Why are they important while transcoding [decode-encode] videos ?

What does this code bit do in ffmpeg.c , what is its purpose?

01562    ist->next_pts = ist->pts = picture.best_effort_timestamp;
01563    if (ist->st->codec->time_base.num != 0) {
01564        int ticks= ist->st->parser ? ist->st->parser->repeat_pict+1 : ist->st->codec->ticks_per_frame;
01565        ist->next_pts += ((int64_t)AV_TIME_BASE *
01566                         ist->st->codec->time_base.num * ticks) /
01567                         ist->st->codec->time_base.den;
01568    }

Solution

Those are the decoding time stamp (DTS) and presentation time stamp (PTS). You can find an explanation here inside a tutorial.

So let's say we had a movie, and the frames were displayed like: I B B P. Now, we need to know the information in P before we can display either B frame. Because of this, the frames might be stored like this: I P B B. This is why we have a decoding timestamp and a presentation timestamp on each frame. The decoding timestamp tells us when we need to decode something, and the presentation time stamp tells us when we need to display something. So, in this case, our stream might look like this:

   PTS: 1 4 2 3
   DTS: 1 2 3 4
Stream: I P B B

Generally the PTS and DTS will only differ when the stream we are playing has B frames in it.

Comments

[nirvanaswap]

Can you please epand on what a B frame is, and what a P frame is?

[Bart]

@nirvanaswap that is actually explained fairly well here en.wikipedia.org/wiki/Video_compression_picture_types

[nirvanaswap]

I still don't understand. The stream has to be displayed as I B B P, But since P ought to come before B, we store it as I P B B. Hence we decode it in the order of 1 2 3 4, that makes sense. But shouldn't we present it in the order of 1 3 4 2? Why is it 1 4 2 3? When we have consecutive B frames, do these B frames take advantage of the I and P frames alone, or do they use the B frames next to each other too?

[nirvanaswap]

PTS of 1 4 2 3 makes it sound like we're displaying the frames in the order of I B(2) P B(1). Correct me if I'm wrong...

[nirvanaswap]

Quick follow up: As a rule, is the decoding timestamp always in ascending order? It appears so.

[Bart]

No, you're displaying in IBBP order. But because the first B relies on the I before it and the B after it, and subsequently the second B frame relies on the final P frame, to be able to display the first B frame you'll already need the information of the P frame. And that's why the stream is IPBB.

[nirvanaswap]

I'm sorry, I'm lost. Let's say the frames need to be displayed in 1 B(1) B(2) P. B(1) depends on 1 and B(2). I assume B(2) depends only on P. Therefore we should store it as I P B(2) B(1) right? Decoding is in this order, but then presentation should be I B(1) B(2) P, that is, 1 4 3 2, right?

[Bart]

B frames are bidirectional and depend on information before and after it. So B2 depends on both B1 and P. As the text states "we need to know the information in P before we can display either B frame". And therefore the final order you need the data in is IPBB. At least that is how I understand it. More in depth information is probably best requested from someone who is more up to date than I am.

[nirvanaswap]

Well if B1 depends on I and B2, and B2 depends on B1 and P, it sounds like a deadlock to me. Anyway, thanks.

[arielCo]

In case @nirvanaswap or anyone else is still wondering, the frames are stored in dependency order, so for an AVC stream sent as I-P1-B1-B2-..., B2 may rely on B1 but not the other way around, regardless of their display order.

[neevek]

@arielCo In that case, DTS is guaranteed to appear in ascending order, right?

[arielCo]

@neevek Should; ffmpeg complains when it doesn't, eg “Non-monotonous DTS in output stream” (in this case they used -c copy).

[ahmetknk]

Note for confused readers like me: I, P, and B are not arbitrary frame names they stand for Intra-coded frame, Predicted frame, Bi-directional predicted frame, respectively.