Android : How to change Playback Rate of music using OpenSL ES

Question

I am working on a music player in which I need to change tempo (playback speed of music) without changing the pitch.

I'm not able to find any native android class to do so. I tried SoundPool but it doesn't work with large music files and it also doesn't seems to work on many devices. I also tried AudioTrack but again no luck.

Now I am trying android NDK audio example which use OpenSL ES to handle music. Now I just want to add set playback rate feature in this example.

Can anyone show me how do I add change playback rate function in it?

Solution 1

I have solved my problem. Here is my complete native code for OpenSL ES in case of anybody need this :

#include <jni.h>

#include<android/log.h>
// LOCAL_LDLIBS := -L$(SYSROOT)/usr/lib -llog 넣어주세요
#define LOGV(...) __android_log_print(ANDROID_LOG_VERBOSE, "OSLESMediaPlayer", __VA_ARGS__) 
#define LOGD(...) __android_log_print(ANDROID_LOG_DEBUG  , "OSLESMediaPlayer", __VA_ARGS__) 
#define LOGI(...) __android_log_print(ANDROID_LOG_INFO   , "OSLESMediaPlayer", __VA_ARGS__) 
#define LOGW(...) __android_log_print(ANDROID_LOG_WARN   , "OSLESMediaPlayer", __VA_ARGS__) 
#define LOGE(...) __android_log_print(ANDROID_LOG_ERROR  , "OSLESMediaPlayer", __VA_ARGS__) 

// for native audio
#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>

#include <assert.h>
#include <sys/types.h>

// engine interfaces
static SLObjectItf engineObject = NULL;
static SLEngineItf engineEngine;

// URI player interfaces
static SLObjectItf uriPlayerObject = NULL;
static SLPlayItf uriPlayerPlay;
static SLSeekItf uriPlayerSeek;
static SLPlaybackRateItf uriPlaybackRate;

// output mix interfaces
static SLObjectItf outputMixObject = NULL;

// playback rate (default 1x:1000)
static SLpermille playbackMinRate = 500;
static SLpermille playbackMaxRate = 2000;
static SLpermille playbackRateStepSize;

//Pitch
static SLPitchItf uriPlaybackPitch;
static SLpermille playbackMinPitch = 500;
static SLpermille playbackMaxPitch = 2000;

// create the engine and output mix objects
JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_createEngine(
        JNIEnv* env, jclass clazz) {
    SLresult result;

    // create engine
    LOGD("create engine");
    result = slCreateEngine(&engineObject, 0, NULL, 0, NULL, NULL);
    assert(SL_RESULT_SUCCESS == result);

    // realize the engine
    LOGD("realize the engine");
    result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
    assert(SL_RESULT_SUCCESS == result);

    // get the engine interface, which is needed in order to create other objects
    LOGD("get the engine interface");
    result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE,
            &engineEngine);
    assert(SL_RESULT_SUCCESS == result);

    // create output mix, with environmental reverb specified as a non-required interface
    LOGD("create output mix");
    const SLInterfaceID ids[1] = {SL_IID_PLAYBACKRATE};
    const SLboolean req[1] = {SL_BOOLEAN_FALSE};
    result = (*engineEngine)->CreateOutputMix(engineEngine, &outputMixObject, 1,
            ids, req);
    assert(SL_RESULT_SUCCESS == result);

    // realize the output mix
    LOGD("realize the output mix");
    result = (*outputMixObject)->Realize(outputMixObject, SL_BOOLEAN_FALSE);
    assert(SL_RESULT_SUCCESS == result);

}

JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_releaseEngine(
        JNIEnv* env, jclass clazz) {
    // destroy URI audio player object, and invalidate all associated interfaces
    if (uriPlayerObject != NULL) {
        (*uriPlayerObject)->Destroy(uriPlayerObject);
        uriPlayerObject = NULL;
        uriPlayerPlay = NULL;
        uriPlayerSeek = NULL;
    }

    // destroy output mix object, and invalidate all associated interfaces
    if (outputMixObject != NULL) {
        (*outputMixObject)->Destroy(outputMixObject);
        outputMixObject = NULL;
    }

    // destroy engine object, and invalidate all associated interfaces
    if (engineObject != NULL) {
        (*engineObject)->Destroy(engineObject);
        engineObject = NULL;
        engineEngine = NULL;
    }

}

/*
 void OnCompletion(JNIEnv* env, jclass clazz)
 {
 jclass cls = env->GetObjectClass(thiz);
 if (cls != NULL)
 {
 jmethodID mid = env->GetMethodID(cls, "OnCompletion", "()V");
 if (mid != NULL)
 {
 env->CallVoidMethod(thiz, mid, 1234);
 }
 }
 }*/

void playStatusCallback(SLPlayItf play, void* context, SLuint32 event) {
    //LOGD("playStatusCallback");
}

// create URI audio player
JNIEXPORT jboolean Java_com_swssm_waveloop_audio_OSLESMediaPlayer_createAudioPlayer(
        JNIEnv* env, jclass clazz, jstring uri) {
    SLresult result;

    // convert Java string to UTF-8
    const jbyte *utf8 = (*env)->GetStringUTFChars(env, uri, NULL);
    assert(NULL != utf8);

    // configure audio source
    // (requires the INTERNET permission depending on the uri parameter)
    SLDataLocator_URI loc_uri = { SL_DATALOCATOR_URI, (SLchar *) utf8 };
    SLDataFormat_MIME format_mime = { SL_DATAFORMAT_MIME, NULL,
            SL_CONTAINERTYPE_UNSPECIFIED };
    SLDataSource audioSrc = { &loc_uri, &format_mime };

    // configure audio sink
    SLDataLocator_OutputMix loc_outmix = { SL_DATALOCATOR_OUTPUTMIX,
            outputMixObject };
    SLDataSink audioSnk = { &loc_outmix, NULL };

    // create audio player
    const SLInterfaceID ids[2] = { SL_IID_SEEK, SL_IID_PLAYBACKRATE };
    const SLboolean req[2] = { SL_BOOLEAN_FALSE, SL_BOOLEAN_TRUE };
    result = (*engineEngine)->CreateAudioPlayer(engineEngine, &uriPlayerObject,
            &audioSrc, &audioSnk, 2, ids, req);
    // note that an invalid URI is not detected here, but during prepare/prefetch on Android,
    // or possibly during Realize on other platforms
    assert(SL_RESULT_SUCCESS == result);

    // release the Java string and UTF-8
    (*env)->ReleaseStringUTFChars(env, uri, utf8);

    // realize the player
    result = (*uriPlayerObject)->Realize(uriPlayerObject, SL_BOOLEAN_FALSE);
    // this will always succeed on Android, but we check result for portability to other platforms
    if (SL_RESULT_SUCCESS != result) {
        (*uriPlayerObject)->Destroy(uriPlayerObject);
        uriPlayerObject = NULL;
        return JNI_FALSE;
    }

    // get the play interface
    result = (*uriPlayerObject)->GetInterface(uriPlayerObject, SL_IID_PLAY,
            &uriPlayerPlay);
    assert(SL_RESULT_SUCCESS == result);

    // get the seek interface
    result = (*uriPlayerObject)->GetInterface(uriPlayerObject, SL_IID_SEEK,
            &uriPlayerSeek);
    assert(SL_RESULT_SUCCESS == result);

    // get playback rate interface
    result = (*uriPlayerObject)->GetInterface(uriPlayerObject,
            SL_IID_PLAYBACKRATE, &uriPlaybackRate);
    assert(SL_RESULT_SUCCESS == result);

    /*  // get playback pitch interface
     result = (*uriPlayerObject)->GetInterface(uriPlayerObject, SL_IID_PITCH, &uriPlaybackPitch);
     assert(SL_RESULT_SUCCESS == result);*/

    // register callback function
    result = (*uriPlayerPlay)->RegisterCallback(uriPlayerPlay,
            playStatusCallback, 0);
    assert(SL_RESULT_SUCCESS == result);
    result = (*uriPlayerPlay)->SetCallbackEventsMask(uriPlayerPlay,
            SL_PLAYEVENT_HEADATEND); // head at end
    assert(SL_RESULT_SUCCESS == result);

    SLmillisecond msec;
    result = (*uriPlayerPlay)->GetDuration(uriPlayerPlay, &msec);
    assert(SL_RESULT_SUCCESS == result);

    // no loop
    result = (*uriPlayerSeek)->SetLoop(uriPlayerSeek, SL_BOOLEAN_TRUE, 0, msec);
    assert(SL_RESULT_SUCCESS == result);


    SLuint32 capa;
        result = (*uriPlaybackRate)->GetRateRange(uriPlaybackRate, 0,
                &playbackMinRate, &playbackMaxRate, &playbackRateStepSize, &capa);
        assert(SL_RESULT_SUCCESS == result);

        result = (*uriPlaybackRate)->SetPropertyConstraints(uriPlaybackRate,
                        SL_RATEPROP_PITCHCORAUDIO);

                    if (SL_RESULT_PARAMETER_INVALID == result) {
                        LOGD("Parameter Invalid");
                    }
                    if (SL_RESULT_FEATURE_UNSUPPORTED == result) {
                            LOGD("Feature Unsupported");
                        }
                    if (SL_RESULT_SUCCESS == result) {
                        assert(SL_RESULT_SUCCESS == result);
                            LOGD("Success");
                        }
    /*
     result = (*uriPlaybackPitch)->GetPitchCapabilities(uriPlaybackPitch, &playbackMinPitch, &playbackMaxPitch);
     assert(SL_RESULT_SUCCESS == result);*/

    /*
     SLpermille minRate, maxRate, stepSize, rate = 1000;
     SLuint32 capa;
     (*uriPlaybackRate)->GetRateRange(uriPlaybackRate, 0, &minRate, &maxRate, &stepSize, &capa);

     (*uriPlaybackRate)->SetRate(uriPlaybackRate, minRate);
     */
    return JNI_TRUE;
}

JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_releaseAudioPlayer(
        JNIEnv* env, jclass clazz) {
    // destroy URI audio player object, and invalidate all associated interfaces
    if (uriPlayerObject != NULL) {
        (*uriPlayerObject)->Destroy(uriPlayerObject);
        uriPlayerObject = NULL;
        uriPlayerPlay = NULL;
        uriPlayerSeek = NULL;
        uriPlaybackRate = NULL;
    }

}

void setPlayState(SLuint32 state) {
    SLresult result;

    // make sure the URI audio player was created
    if (NULL != uriPlayerPlay) {

        // set the player's state
        result = (*uriPlayerPlay)->SetPlayState(uriPlayerPlay, state);
        assert(SL_RESULT_SUCCESS == result);
    }

}

SLuint32 getPlayState() {
    SLresult result;

    // make sure the URI audio player was created
    if (NULL != uriPlayerPlay) {

        SLuint32 state;
        result = (*uriPlayerPlay)->GetPlayState(uriPlayerPlay, &state);
        assert(SL_RESULT_SUCCESS == result);

        return state;
    }

    return 0;

}

// play
JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_play(JNIEnv* env,
        jclass clazz) {
    setPlayState(SL_PLAYSTATE_PLAYING);
}

// stop
JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_stop(JNIEnv* env,
        jclass clazz) {
    setPlayState(SL_PLAYSTATE_STOPPED);
}

// pause
JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_pause(JNIEnv* env,
        jclass clazz) {
    setPlayState(SL_PLAYSTATE_PAUSED);
}

// pause
JNIEXPORT jboolean Java_com_swssm_waveloop_audio_OSLESMediaPlayer_isPlaying(
        JNIEnv* env, jclass clazz) {
    return (getPlayState() == SL_PLAYSTATE_PLAYING);
}

// set position
JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_seekTo(
        JNIEnv* env, jclass clazz, jint position) {
    if (NULL != uriPlayerPlay) {

        //SLuint32 state = getPlayState();
        //setPlayState(SL_PLAYSTATE_PAUSED);

        SLresult result;

        result = (*uriPlayerSeek)->SetPosition(uriPlayerSeek, position,
                SL_SEEKMODE_FAST);
        assert(SL_RESULT_SUCCESS == result);

        //setPlayState(state);
    }

}

// get duration
JNIEXPORT jint Java_com_swssm_waveloop_audio_OSLESMediaPlayer_getDuration(
        JNIEnv* env, jclass clazz) {
    if (NULL != uriPlayerPlay) {

        SLresult result;

        SLmillisecond msec;
        result = (*uriPlayerPlay)->GetDuration(uriPlayerPlay, &msec);
        assert(SL_RESULT_SUCCESS == result);

        return msec;
    }

    return 0.0f;
}

// get current position
JNIEXPORT jint Java_com_swssm_waveloop_audio_OSLESMediaPlayer_getPosition(
        JNIEnv* env, jclass clazz) {
    if (NULL != uriPlayerPlay) {

        SLresult result;

        SLmillisecond msec;
        result = (*uriPlayerPlay)->GetPosition(uriPlayerPlay, &msec);
        assert(SL_RESULT_SUCCESS == result);

        return msec;
    }

    return 0.0f;
}

//llllllllllllllllllll

JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_setPitch(
        JNIEnv* env, jclass clazz, jint rate) {
    if (NULL != uriPlaybackPitch) {
        SLresult result;

        result = (*uriPlaybackPitch)->SetPitch(uriPlaybackPitch, rate);
        assert(SL_RESULT_SUCCESS == result);
    }
}

JNIEXPORT void Java_com_swssm_waveloop_audio_OSLESMediaPlayer_setRate(
        JNIEnv* env, jclass clazz, jint rate) {
    if (NULL != uriPlaybackRate) {
        SLresult result;

        result = (*uriPlaybackRate)->SetRate(uriPlaybackRate, rate);
            assert(SL_RESULT_SUCCESS == result);


    }
}

JNIEXPORT jint Java_com_swssm_waveloop_audio_OSLESMediaPlayer_getRate(
        JNIEnv* env, jclass clazz) {
    if (NULL != uriPlaybackRate) {
        SLresult result;

        SLpermille rate;
        result = (*uriPlaybackRate)->GetRate(uriPlaybackRate, &rate);
        assert(SL_RESULT_SUCCESS == result);

        return rate;
    }

    return 0;
}

// create URI audio player
JNIEXPORT jboolean Java_com_swssm_waveloop_audio_OSLESMediaPlayer_setLoop(
        JNIEnv* env, jclass clazz, jint startPos, jint endPos) {
    SLresult result;

    result = (*uriPlayerSeek)->SetLoop(uriPlayerSeek, SL_BOOLEAN_TRUE, startPos,
            endPos);
    assert(SL_RESULT_SUCCESS == result);

    return JNI_TRUE;
}

// create URI audio player
JNIEXPORT jboolean Java_com_swssm_waveloop_audio_OSLESMediaPlayer_setNoLoop(
        JNIEnv* env, jclass clazz) {
    SLresult result;
    if (NULL != uriPlayerSeek) {
        // enable whole file looping
        result = (*uriPlayerSeek)->SetLoop(uriPlayerSeek, SL_BOOLEAN_TRUE, 0,
                SL_TIME_UNKNOWN);
        assert(SL_RESULT_SUCCESS == result);

    }
    return JNI_TRUE;
}

Just compile it using ndk-build command and use it. If anybody get success in changing pitch then please tell me the solution.

Here is android.mk file

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE    := audio-tools

LOCAL_SRC_FILES := OSLESMediaPlayer.c


LOCAL_CFLAGS := -DHAVE_CONFIG_H -DFPM_ARM -ffast-math -O3

LOCAL_LDLIBS    += -lOpenSLES -llog

include $(BUILD_SHARED_LIBRARY)

and Application.mk file

APP_STL := gnustl_static
APP_CPPFLAGS += -fexceptions -frtti
APP_ABI := armeabi armeabi-v7a

And wrapper class, you can use its function directly in your project

package com.swssm.waveloop.audio;
public class OSLESMediaPlayer {
    public native void createEngine();
    public native void releaseEngine();
    public native boolean createAudioPlayer(String uri);
    public native void releaseAudioPlayer();
    public native void play();
    public native void stop();
    public native void pause();
    public native boolean isPlaying();

    public native void seekTo(int position);
    public native int getDuration();
    public native int getPosition();

    public native void setPitch(int rate);

    public native void setRate(int rate);
    public native int getRate();

    public native void setLoop( int startPos, int endPos );
    public native void setNoLoop();


    public interface OnCompletionListener {
        public void OnCompletion();
    }

    private OnCompletionListener mCompletionListener;
    public void SetOnCompletionListener( OnCompletionListener listener )
    {
        mCompletionListener = listener;
    }


    private void OnCompletion()
    {
        mCompletionListener.OnCompletion();

        int position = getPosition();
        int duration = getDuration();
        if( position != duration )
        {
            int a = 0;

        }
        else
        {
            int c = 0;

        }
    }
}

Solution 2

This may be of some help (taken from the NDK OpenSL documentation):

Playback rate

The supported playback rate range(s) and capabilities may vary depending on the platform version and implementation, and so should be determined at runtime by querying with PlaybackRate::GetRateRange or PlaybackRate::GetCapabilitiesOfRate.

That said, some guidance on typical rate ranges may be useful: In Android 2.3 a single playback rate range from 500 per mille to 2000 per mille inclusive is typically supported, with property SL_RATEPROP_NOPITCHCORAUDIO. In Android 4.0 the same rate range is typically supported for a data source in PCM format, and a unity rate range for other formats.

Comments

[Harsh]

Thanx for sharing your knowledge . Can you tell me that will it work with .Mp3 files. Can I use this lib and code to change the tempo of .mp3 files. Thanx in advanced vipul

[Vipul Purohit]

Yes. It will work with all kind of MP3, WAV etc. file formats. But unfortunately tempo change features will not work on ICS. I dont know why but this feature is been removed from OpenSL ES in Android ICS.

[Harsh]

Thanx . IN your code snippet you are including 4 files (#include <SLES/OpenSLES.h> #include <SLES/OpenSLES_Android.h> #include <assert.h> #include <sys/types.h>) Where can i find these files.

[Vipul Purohit]

These are OpenSL ES is an inbuilt android library so you don/t have to include any external files for using them.

[Harsh]

I followed the your suggestions but was facing problems the ndk build command was not executing. So i searched about opensl es and found the is supported on API level 9 and above(check here link ) . But i am developing my app on Api level 8. What should i do ?

[Vipul Purohit]

There is no way. You must have to use API level 9 as OpenSL ES was introduced in API level 9. If you dont want to use OpenSL ES then use FFMPEG lib. But Its much more complex the using OpenSL ES

[Harsh]

Hi again vipul I found only you on stack overflow who is driving me towards right direction ... With your advice i changed my mind to use ffmpeg . I am tryng to compile ffmpeg with ndk and as this is my 1st time with Android ndk it looks quite complex me to compile ffmpeg for android. And are you sure this ffmpeg lib can help me to change tempo of mp3 files . If Yes then how . Can you help

[Harsh]

I forgot to tell you that i am trying to compile it on windows

[Harsh]

Hi vipul I chnged version of my app to 2.3 to use opensl es . And with your help i have compiled opensl es .(With your code shared above) Now please can you tell me how to use native method setrate to change the Tempo of my mp3 file , which is placed in raw folder. Thanx Vipul for helping me.

[Vipul Purohit]

Hello, First use createEngine() method to create engine. Then use createAudioPlayer(FILE PATH); to open file and then you can use setPitch(int rate); or setRate(int rate); function along with play();.

[Ram]

Hello Vipul, I need to set Reverb and Tone [Play back rate ] to the player file,So please guide me I am new to these stuff. I have tried the code : eReverb = new EnvironmentalReverb(0, 0); eReverb.setDecayHFRatio((short) 1000); eReverb.setDecayTime(10000); eReverb.setDensity((short) 1000); eReverb.setDiffusion((short) 1000); eReverb.setReverbLevel((short) 1000); eReverb.setReverbDelay(100); eReverb.setEnabled(true); eReverb.setReflectionsDelay(100); mMediaPlayer.attachAuxEffect(eReverb.getId()); but not working.....Is it correct ? I have tried with