Reading audio samples via AVAssetReader

Question

How do you read audio samples via AVAssetReader? I've found examples of duplicating or mixing using AVAssetReader, but those loops are always controlled by the AVAssetWriter loop. Is it possible just to create an AVAssetReader and read through it, getting each sample and throwing the int32 of each audio sample into an array?

Thanks.

Solution 1

To expand on @amrox's answer, you can get an AudioBufferList from the CMBlockBufferRef, e.g.

CMItemCount numSamplesInBuffer = CMSampleBufferGetNumSamples(buffer);

AudioBufferList audioBufferList;

CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
      buffer,
      NULL,
      &audioBufferList,
      sizeof(audioBufferList),
      NULL,
      NULL,
      kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
      &buffer
    );

for (int bufferCount=0; bufferCount < audioBufferList.mNumberBuffers; bufferCount++) {
  SInt16* samples = (SInt16 *)audioBufferList.mBuffers[bufferCount].mData;
  for (int i=0; i < numSamplesInBuffer; i++) {
    // amplitude for the sample is samples[i], assuming you have linear pcm to start with
  }
}

//Release the buffer when done with the samples 
//(retained by CMSampleBufferGetAudioBufferListWithRetainedblockBuffer)
CFRelease(buffer); 

Solution 2

AVAssetReader *reader   = [[AVAssetReader alloc] initWithAsset:asset error:&error];
AVAssetTrack  *track    = [[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
NSDictionary  *settings = @{ AVFormatIDKey : [NSNumber numberWithInt:kAudioFormatLinearPCM] };

AVAssetReaderTrackOutput *readerOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:track
                                                                                    outputSettings:settings];

[reader addOutput:readerOutput]; 
[reader startReading];

CMSampleBufferRef sample = [readerOutput copyNextSampleBuffer];

while ( sample )
{
   sample = [readerOutput copyNextSampleBuffer];

    if ( ! sample )
    {
       continue;
    }

    CMBlockBufferRef buffer = CMSampleBufferGetDataBuffer(sample);

    size_t  lengthAtOffset;
    size_t  totalLength;
    char   *data;

    if ( CMBlockBufferGetDataPointer( buffer, 0, &lengthAtOffset, &totalLength, &data ) != noErr )
    {
        NSLog(@"error!");
        break;
    }

    // do something with data...

    CFRelease(sample);
}

Solution 3

The answers here are not generic. The call to CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer might fail when the AudioBufferList needs to be sized differently. When having non-intererleaved samples as example.

The correct way is to call CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer twice. The first call queries the size needed for the AudioBufferList and second one actually fills the AudioBufferList.

size_t bufferSize = 0;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
    sampleBuffer,
    &bufferSize,
    NULL,
    0,
    NULL,
    NULL,
    0,
    NULL
);

AudioBufferList *bufferList = malloc(bufferSize);
CMBlockBufferRef blockBuffer = NULL;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
    sampleBuffer,
    NULL,
    bufferList,
    bufferSize,
    NULL,
    NULL,
    kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
    &blockBuffer
);

// handle audio here

free(bufferList);
CFRelease(blockBuffer);

In a real world example you must perform error handling and also you should not malloc every frame, instead cache the AudioBufferList.

Comments

[Eric Christensen]

Thanks @amrox. This helps some. But I think it's missing adding the output and starting the reading. That can just be added with: [reader addOutput:readerOutput]; [reader startReading]; after you create readerOutput, right? Once I add that, this doesn't read through a whole audio file, but only once or twice. Is that how it's supposed to work? If so, I don't understand how this gets me to the individual samples. Each sample in an audio file is an int, with the amount and range defined by the bit depth and sample rate. How can I get to those ints? Thanks again.

[bluevoid]

They're in that char* data. Set more keys in the settings dictionary if you want more control: AVNumberOfChannelsKey, AVLinearPCMBitDepthKey. And don't forget to award the bounty to amrox.

[Eric Christensen]

Thanks so much! That was the part that I was missing.

[Dex]

When I use this method my sound gets choppy, when I use @amrox's method, which uses a char*, everything works fine when altering the volume by doing something like samples[i] *= 0.5;Any reason for this?

[Chan Jing Hong]

Is it possible to get the samples and then write it into another audio file? I'm trying to reverse an audio file. Please take a look at my question. Any help would be appreciated.

[omarojo]

but then how do you play that audio from the speakers ?

[JCutting8]

Swift 4 I get the error "Type 'AudioBuffer' has no subscript members" when declaring the variable "samples" in the for loop.