Extract iPod Library raw PCM samples and play with sound effects

Question

I am trying to extract raw PCM samples from an MP3 in the iPod Library so that I can play the song and manipulate the pitch, tempo, and apply sound effects (such as filters). I have already gone down the route of AVPlayer and AVAudioPlayer which both do not allow very much control over the playback at all.

The code below is as far as I have gotten with this. I am at a point now where I do not know what to do with the CMSampleBufferRef's in my while loop because I do not know which framework to use in order to playback the audio and apply such effects.

Any idea what would be the best approach to achieve this? I have looked at cases where the file is converted using an AVAssetWriter but this is not going to cut it for me because the process is too time consuming. Surely I can just read the PCM samples into memory for playback without having to write them to disk first?

NB: I know the code below references an mp3 within the project but I am aware that this approach will work the same as if I were pulling an NSURL from the MPMediaPropertyAssetURL

-(IBAction)loadTrack:(id)sender {

 NSString *songPath = [[NSBundle mainBundle] pathForResource:@"Smooth_Sub Focus_192" ofType:@"mp3"];
 NSURL *assetURL = [[NSURL alloc] initFileURLWithPath:songPath];

 AVURLAsset *songAsset = [AVURLAsset URLAssetWithURL:assetURL options:nil];

 NSError *assetError = nil;
 AVAssetReader *assetReader = [[AVAssetReader assetReaderWithAsset:songAsset
                error:&assetError] retain];
 if (assetError) {
  NSLog (@"Error: %@", assetError);
  return;
 }

 AVAssetReaderOutput *assetReaderOutput = [[AVAssetReaderAudioMixOutput assetReaderAudioMixOutputWithAudioTracks:songAsset.tracks
                           audioSettings: nil] retain];
 if (![assetReader canAddOutput:assetReaderOutput]) {
  NSLog (@"Incompatible Asser Reader Output");
  return;
 }

 [assetReader addOutput: assetReaderOutput];
 [assetReader startReading];

 CMSampleBufferRef nextBuffer;
 while (nextBuffer = [assetReaderOutput copyNextSampleBuffer]) {
  /* What Do I Do Here? */
 }

 [assetReader release];
 [assetReaderOutput release];

}

Solution 1

I'm doing something similar in my own code. The following method returns some NSData for a AVURLAsset:

- (NSData *)extractDataForAsset:(AVURLAsset *)songAsset {

    NSError * error = nil;
    AVAssetReader * reader = [[AVAssetReader alloc] initWithAsset:songAsset error:&error];

    AVAssetTrack * songTrack = [songAsset.tracks objectAtIndex:0];
    AVAssetReaderTrackOutput * output = [[AVAssetReaderTrackOutput alloc] initWithTrack:songTrack outputSettings:nil];
    [reader addOutput:output];
    [output release];

    NSMutableData * fullSongData = [[NSMutableData alloc] init];
    [reader startReading];

    while (reader.status == AVAssetReaderStatusReading){

        AVAssetReaderTrackOutput * trackOutput = (AVAssetReaderTrackOutput *)[reader.outputs objectAtIndex:0];
        CMSampleBufferRef sampleBufferRef = [trackOutput copyNextSampleBuffer];

        if (sampleBufferRef){
            CMBlockBufferRef blockBufferRef = CMSampleBufferGetDataBuffer(sampleBufferRef);

            size_t length = CMBlockBufferGetDataLength(blockBufferRef);
            UInt8 buffer[length];
            CMBlockBufferCopyDataBytes(blockBufferRef, 0, length, buffer);

            NSData * data = [[NSData alloc] initWithBytes:buffer length:length];
            [fullSongData appendData:data];
            [data release];

            CMSampleBufferInvalidate(sampleBufferRef);
            CFRelease(sampleBufferRef);
        }
    }

    if (reader.status == AVAssetReaderStatusFailed || reader.status == AVAssetReaderStatusUnknown){
        // Something went wrong. Handle it.
    }

    if (reader.status == AVAssetReaderStatusCompleted){
        // You're done. It worked.
    }

    [reader release];

    return [fullSongData autorelease];
}

I would recommend doing this on a background thread because it's time consuming.

A drawback to this method is that the whole song is loaded into memory, which is of course limited.

Solution 2

Additionally to Tom Irving's Answer, I suggest replacing

       UInt8 buffer[length];
       CMBlockBufferCopyDataBytes(blockBufferRef, 0, length, buffer);

        NSData * data = [[NSData alloc] initWithBytes:buffer length:length];

with

        NSMutableData * data = [[NSMutableData alloc] initWithLength:length];
        CMBlockBufferCopyDataBytes(blockBufferRef, 0, length, data.mutableBytes);

which avoids double handling the samples, and reduces memory usage overhead.

alternatively, you can wrap [NSMutableData dataWithLength:length] in an auto release pool as demonstrated in this answer to an unrelated but similar question.

Solution 3

I think you want this in there to ensure its PCM...

NSDictionary* outputSettingsDict = [[NSDictionary alloc] initWithObjectsAndKeys:

                        [NSNumber numberWithInt:kAudioFormatLinearPCM],AVFormatIDKey,
                   //     [NSNumber numberWithInt:44100.0],AVSampleRateKey, /*Not Supported*/
                   //     [NSNumber numberWithInt: 2],AVNumberOfChannelsKey,    /*Not Supported*/

                        [NSNumber numberWithInt:16],AVLinearPCMBitDepthKey,
                        [NSNumber numberWithBool:NO],AVLinearPCMIsBigEndianKey,
                        [NSNumber numberWithBool:NO],AVLinearPCMIsFloatKey,
                        [NSNumber numberWithBool:NO],AVLinearPCMIsNonInterleaved,

                        nil];

AVAssetReaderTrackOutput* output = [[AVAssetReaderTrackOutput alloc] initWithTrack:songTrack outputSettings:outputSettingsDict];

Comments

[Dino]

Excellent. This gets me half way there. I've been doing some research on audio frameworks and it seems that the only framework that suits my requirements is OpenAL. With a steep learning curve this is slightly offputting but could I use the NSData with something like Finch?

[Tom Irving]

I've never used Finch myself, so I'm unsure if it directly accepts NSData.

[Tom Irving]

Fantastic - when I put it together I was just trying to get it working, hadn't started on optimisation yet. My biggest annoyance is that I can't seem to find a way to get the full size of the song in bytes before reading the whole thing into memory.

[Tom Irving]

Obviously I could read it once, without copying anything, just getting the length for each sample and sum them, but it seems too messy.

[Larme]

FYI: I tried it, and I got EXC_BAD_ACCESS... I could remove it by checking the OSStatus returned by CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer before doing the for.