What is the difference between AV_SAMPLE_FMT_S16P and AV_SAMPLE_FMT_S16?

Question

What happens when you do a conversion from AV_SAMPLE_FMT_S16P to AV_SAMPLE_FMT_S16? How is the AVFrame structure going to contain the planar and non-planar data?

Solution

AV_SAMPLE_FMT_S16P is planar signed 16 bit audio, i.e. 2 bytes for each sample which is same for AV_SAMPLE_FMT_S16.

The only difference is in AV_SAMPLE_FMT_S16 samples of each channel are interleaved i.e. if you have two channel audio then the samples buffer will look like

c1 c2 c1 c2 c1 c2 c1 c2...

where c1 is a sample for channel1 and c2 is sample for channel2.

while for one frame of planar audio you will have something like

c1 c1 c1 c1 .... c2 c2 c2 c2 ..

---

now how is it stored in AVFrame:

• for planar audio:

data[i] will contain the data of channel i (assuming channel 0 is first channel).

however if you have more channels than 8, then data for rest of the channels can be found in extended_data attribute of AVFrame.

• for non-planar audio

data[0] will contain the data for all channels in an interleaved manner.

Comments

[MarcusJ]

if the audio data is stored in data[0] for non-planar formats, does that mean we have to read that array index 16/24 bits at a time, until NULL? Like, each sample doesn't get it's own array index, each index is associated with a channel, instead? so it's just a single long list o samples...

[DuBistKomisch]

I assume c1 c1 c2 c2 must refer to the bytes in the buffer, not the samples. Should either change it to c1 c2 c1 c2 for samples, or update the text to say bytes.

[Darkwonder]

Thank you for this answer. I have been struggling several days with FFmpegs audio data and CMSampleBuffer creation via CMBlockBufferCreation. Your answer was the breakthrough I needed. Would it be possible to upvote the answer more than once, I would upvote it to Inf+.