Android Speech Recognition as a service on Android 4.1 & 4.2

Question

I have managed to get continuous speech recognition working (using the SpeechRecognizer class) as a service on all Android versions up to 4.1. My question concerns getting it working on versions 4.1 and 4.2 as it is known there is a problem in that the API doesn't do as documented in that a few seconds after voice recognition is started, if no voice input has been detected then it's as if the speech recogniser dies silently. (http://code.google.com/p/android/issues/detail?id=37883)

I have found a question which proposes a work-around to this problem (Voice Recognition stops listening after a few seconds), but I am unsure as how to implement the Handler required for this solution. I am aware of the 'beep' that will happen every few seconds that this workaround will cause, but getting continuous voice recognition is more important for me.

If anyone has any other alternative workarounds then I'd like to hear those too.

Solution 1

This is a work around for android version 4.1.1.

public class MyService extends Service
{
    protected AudioManager mAudioManager; 
    protected SpeechRecognizer mSpeechRecognizer;
    protected Intent mSpeechRecognizerIntent;
    protected final Messenger mServerMessenger = new Messenger(new IncomingHandler(this));

    protected boolean mIsListening;
    protected volatile boolean mIsCountDownOn;
    private boolean mIsStreamSolo;

    static final int MSG_RECOGNIZER_START_LISTENING = 1;
    static final int MSG_RECOGNIZER_CANCEL = 2;

    @Override
    public void onCreate()
    {
        super.onCreate();
        mAudioManager = (AudioManager) getSystemService(Context.AUDIO_SERVICE); 
        mSpeechRecognizer = SpeechRecognizer.createSpeechRecognizer(this);
        mSpeechRecognizer.setRecognitionListener(new SpeechRecognitionListener());
        mSpeechRecognizerIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
        mSpeechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                                         RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
        mSpeechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE,
                                         this.getPackageName());
    }

    protected static class IncomingHandler extends Handler
    {
        private WeakReference<MyService> mtarget;

        IncomingHandler(MyService target)
        {
            mtarget = new WeakReference<MyService>(target);
        }


        @Override
        public void handleMessage(Message msg)
        {
            final MyService target = mtarget.get();

            switch (msg.what)
            {
                case MSG_RECOGNIZER_START_LISTENING:

                    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN)
                    {
                        // turn off beep sound  
                        if (!mIsStreamSolo)
                        {
                            mAudioManager.setStreamSolo(AudioManager.STREAM_VOICE_CALL, true);
                            mIsStreamSolo = true;
                        }
                    }
                     if (!target.mIsListening)
                     {
                         target.mSpeechRecognizer.startListening(target.mSpeechRecognizerIntent);
                         target.mIsListening = true;
                        //Log.d(TAG, "message start listening"); //$NON-NLS-1$
                     }
                     break;

                 case MSG_RECOGNIZER_CANCEL:
                    if (mIsStreamSolo)
                   {
                        mAudioManager.setStreamSolo(AudioManager.STREAM_VOICE_CALL, false);
                        mIsStreamSolo = false;
                   }
                      target.mSpeechRecognizer.cancel();
                      target.mIsListening = false;
                      //Log.d(TAG, "message canceled recognizer"); //$NON-NLS-1$
                      break;
             }
       } 
    } 

    // Count down timer for Jelly Bean work around
    protected CountDownTimer mNoSpeechCountDown = new CountDownTimer(5000, 5000)
    {

        @Override
        public void onTick(long millisUntilFinished)
        {
            // TODO Auto-generated method stub

        }

        @Override
        public void onFinish()
        {
            mIsCountDownOn = false;
            Message message = Message.obtain(null, MSG_RECOGNIZER_CANCEL);
            try
            {
                mServerMessenger.send(message);
                message = Message.obtain(null, MSG_RECOGNIZER_START_LISTENING);
                mServerMessenger.send(message);
            }
            catch (RemoteException e)
            {

            }
        }
    };

    @Override
    public void onDestroy()
    {
        super.onDestroy();

        if (mIsCountDownOn)
        {
            mNoSpeechCountDown.cancel();
        }
        if (mSpeechRecognizer != null)
        {
            mSpeechRecognizer.destroy();
        }
    }

    protected class SpeechRecognitionListener implements RecognitionListener
    {

        @Override
        public void onBeginningOfSpeech()
        {
            // speech input will be processed, so there is no need for count down anymore
            if (mIsCountDownOn)
            {
                mIsCountDownOn = false;
                mNoSpeechCountDown.cancel();
            }               
            //Log.d(TAG, "onBeginingOfSpeech"); //$NON-NLS-1$
        }

        @Override
        public void onBufferReceived(byte[] buffer)
        {

        }

        @Override
        public void onEndOfSpeech()
        {
            //Log.d(TAG, "onEndOfSpeech"); //$NON-NLS-1$
         }

        @Override
        public void onError(int error)
        {
            if (mIsCountDownOn)
            {
                mIsCountDownOn = false;
                mNoSpeechCountDown.cancel();
            }
             mIsListening = false;
             Message message = Message.obtain(null, MSG_RECOGNIZER_START_LISTENING);
             try
             {
                    mServerMessenger.send(message);
             }
             catch (RemoteException e)
             {

             }
            //Log.d(TAG, "error = " + error); //$NON-NLS-1$
        }

        @Override
        public void onEvent(int eventType, Bundle params)
        {

        }

        @Override
        public void onPartialResults(Bundle partialResults)
        {

        }

        @Override
        public void onReadyForSpeech(Bundle params)
        {
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN)
            {
                mIsCountDownOn = true;
                mNoSpeechCountDown.start();

            }
            Log.d(TAG, "onReadyForSpeech"); //$NON-NLS-1$
        }

        @Override
        public void onResults(Bundle results)
        {
            //Log.d(TAG, "onResults"); //$NON-NLS-1$

        }

        @Override
        public void onRmsChanged(float rmsdB)
        {

        }

    }
}

02/16/2013 - Fix beep sound if you use Text To Speech in your app make sure to turn off Solo stream in onResults

Solution 2

If you really want to implement continuous listening without internet connection you need to consider third-party packages, one of them is CMUSphinx, check Pocketsphinx android demo for example how to listen for keyword efficiently in offline and react on the specific commands like a key phrase "oh mighty computer". The code to do that is simple:

you create a recognizer and just add keyword spotting search:

recognizer = defaultSetup()
        .setAcousticModel(new File(modelsDir, "hmm/en-us-semi"))
        .setDictionary(new File(modelsDir, "lm/cmu07a.dic"))
        .setKeywordThreshold(1e-5f)
        .getRecognizer();

recognizer.addListener(this);
recognizer.addKeywordSearch(KWS_SEARCH_NAME, KEYPHRASE);
switchSearch(KWS_SEARCH_NAME);

and define a listener:

@Override
public void onPartialResult(Hypothesis hypothesis) {
    String text = hypothesis.getHypstr();
    if (text.equals(KEYPHRASE))
      //  do something
} 

Solution 3

For any of you who are trying to silence the beep sound, regrading the @HoanNguyen answer which is very good but be careful as said in the api set setStreamSolo is cumulative so if there is in error in the speech recognition and on error is called(for example no internet connection) then setStremSolo true is called again and again which will result in your app silencing the whole phone (very bad)! the solution to that is to add the setStremMute(false) to the speechRecognizer onError.

Solution 4

check out my demo app : https://github.com/galrom/ContinuesVoiceRecognition

I recommand to use both PockeySphix and SpeechRecognizer.

View more solutions

Comments

[scriptocalypse]

This seems to work, though I did not implement the mute methods inside of a service, but elsewhere.

[Hoan Nguyen]

I wrote and tested the countdown timer as well as muting the beep sound yesterday. It work fine, however there may be a need to synchronize mIsCountDownOn instead of just declaring it volatile.

[Alex B]

I'm running this on a 4.1.2 just by creating a Main Activity that has a button that starts the service, but out of all the different methods in the Service file, only onCreate() and IncomingHandler(MyService target) are actually hit. And then nothing happens. Is this part of the 4.1 and 4.2 problem?

[Hoan Nguyen]

You have to send a START LISTENING message say either onCreate or in onStartCommand

[Hoan Nguyen]

Add the setRecognitionListener in onCreate.

[Kgrover]

@HoanNguyen Hi, I'm new to services. How would I start this service from an Activity (on a button click, for example)? Could you give a simple code snippet? Thanks!

[Hoan Nguyen]

startService(YourClass.this, MyService.class);

[Kgrover]

@HoanNguyen I see, and would that automatically trigger the handler to start listening?

[Hoan Nguyen]

No, you can override onStartCommand and send a MSG_RECOGNIZER_START_LISTENING message

[rmooney]

For some reason, I get target.mSpeechRecognizerIntent and target.mAudioManager are null (causing exceptions) inside the incoming handler. Has anyone encountered this? The onCreate function definitely runs first and instantiates them, but when target is created, those objects are null.

[Hoan Nguyen]

Did you copy the exact code? If so, check if mAudioManager and mSpeechRecognizerIntent are null in onCreate.

[rmooney]

Yes I copied the code exactly, but Eclipse required that I add the unimplemented onBind method. mAudioManager and mSpeechRecognizerIntent are NOT null in onCreate. Everything seems to work fine until I get a null exception trying to access mAudioManager.

[Hoan Nguyen]

If target.mAudioManager is null then it should be null from the beginning. Check to see if target is null.

[rmooney]

target is not null but all the objects inside it are.

[Hoan Nguyen]

That is pretty strange, you should post it as a question with all the relevant codes.

[rmooney]

Thanks for your help. Posted here: stackoverflow.com/questions/18039429/…

[qwr]

@HoanNguyen Hi .I wanted to know if it is possible to use your service as continuous recognition until user press stop button .and from where you got such timing 5000 for countdown . will it make recognition skip some phrases that user uttered

[Hoan Nguyen]

Yes you can by stopping your service. I used 5 seconds because I tested a little and before 5 seconds JB won't cause error. After 5 seconds I call cancel and startListening right away, so unless the user starts speaking in the microseconds gap there should be no problem.

[Mike6679]

I cannot get this code to start listening for speech. onCreate() in the service is called, but onReadyForSpeech() is never called. How exactly do you get speech recognition to start listening?...I tried mSpeechRecognizer.startListening() in onCreate() & also overrided onStart and tried mSpeechRecognizer.startListening() there but speech recognition never starts.

[Hoan Nguyen]

Override onStartCommand and send a MSG_RECOGNIZER_START_LISTENING message

[Mike6679]

I tried that like so: Message msg = new Message(); msg.what = MSG_RECOGNIZER_START_LISTENING; mServerMessenger.send(msg); I'm calling this service from an activity.

[Mike6679]

Ok Here it is: stackoverflow.com/questions/18605513/…

[Mike6679]

Anyone run into slow speech response time on <= ICS? See here: stackoverflow.com/questions/21424723/…

[Pedrum]

Is there a way to make it start listening again after the speech is recognized ?

[Hoan Nguyen]

Send MSG_RECOGNIZER_START_LISTENING after you handled the result.

[daleyjem]

I had to throw the mServerMessenger.send(msg) into a try/catch. @HoanNguyen, thanks for posting this btw... but have you any insight on what could be causing the mAudioManager to not mute on 4.1.2? I've put a breakpoint and seen that it isn't null, but alas, the beep sound.

[daleyjem]

Found my own answer. If anyone can't get the audio muted, you might have success with this: target.mAudioManager.setStreamMute(AudioManager.STREAM_MUSIC‌​, true);

[Pedrum]

@daleyjem that's still wrong because the whole audio system of the device would go off!!

[daleyjem]

@Pedrum, that's true that all sound would be muted, but in all serious... saying "that's wrong" doesn't make my LG Optimus G respect any other 'beep' mute technique.

[daleyjem]

@HoanNguyen.. still didn't prevent the beep for me.

[Pedrum]

there's always a better way in programming.. ill keep tryinnn :)

[DiegoSahagun]

Is this supposed to work on Android < 4.1.1? I am having issues

[Atlas91]

Sorry but in this way what exactly happen? You can decide a keyword as "hello" and then in the onPartialResult only if i say the keyword the tts starts?

[Hoan Nguyen]

Very good catch. I implements a cancel() function which I called in onError which among other thing unset the stream solo. So I never have problem.

[Rajesh Sharma]

HI Please send demo code, how to use it services in activity and how to perform action regarding text. I am new in Android development.

[Glenn Bech]

I believe the use of WeakReference in this code is flawed. A call to "get" on a WeakReference will return null if the object is garbage collected. The rationale for using a WeakReference in this case is also (imho) unclear and makes the example less readable.

[aditya]

can you please me out with getting this service working? i have posted relevant questions and code here : stackoverflow.com/questions/26321969/…

[Dante]

First of all, it is a nice workaround. I know this post is old... I am having an issue with my media player being muted when the Speech Recognition service has started... Is there anyway I can avoid it muting my media player? Thanks

[Hoan Nguyen]

Instead of setStreamSolo try setStreamMute.

[Dante]

I just tried mAudioManager.setStreamMute(AudioManager.STREAM_VOICE_CALL, true); but it didn't mute the beep. My GN2 is on 4.1.1 and I have GS3 is on 4.1.2. I recently updated Google service so that I can downloaded the offline Speech Recognition. Thanks

[Hoan Nguyen]

Try some other streams like STREAM_SYSTEM

[Dante]

Ive tried all STREAM_... but none of them worked except STREAM_MUSIC which also mutes my media player... STREAM_SYSTEM was used to work before I updated Google service for Offline Speech Recognition. Google has changed STREAM_SYSTEM to STREAM_MUSIC for beep sound for some odd reasons... Way to go Google!

[Hoan Nguyen]

I don't think there is any thing you can do about this.

[Ronen Rabinovici]

@HoanNguyen - doesn't this answer result in a gap of around 200-500 mili seconds of no-listening (between one end of speech to one ready for speech)?

[Hoan Nguyen]

What do you mean by this answer? The mute sound or the continuous speech recognizer in a service?

[Ronen Rabinovici]

@HoanNguyen - continuous speech recognizer in a service.

[Hoan Nguyen]

Yes but I process the result and then do something else before I call start listening again. I also prompt the user to speak only when onReadyForSpeech is called, so I have no problem at all.

[Ruchir Baronia]

Will it work even if the screen is off? Also, is KEYPHRASE a string?

[Nikolay Shmyrev]

1) yes, it works when screen is off, though you need to add wakeup lock to prevent your phone from sleep. 2) yes, it is arbitrary string.

[Hoan Nguyen]

It should work when screen if off. Change isStreamSolo to target.isStreamSolo in the IncomingHandler class. However turning off the beep does not work anymore.

[Ruchir Baronia]

I am trying to do a simple keyword spotting in my service, so that I can listen for the word "hello". Unfortunately it is not working. Please help me: stackoverflow.com/questions/35388720/…

[Ruchir Baronia]

Hey I am trying to do a simple keyword spotting in my service, so that I can listen for the word "hello". Unfortunately it is not working. Please help me: stackoverflow.com/questions/35388720/…

[Ruchir Baronia]

Hey I am trying to do a simple keyword spotting in my service, so that I can listen for the word "hello". Unfortunately it is not working. Please help me: stackoverflow.com/questions/35388720/…

[Gal Rom]

@RuchirBaronia , have you checked my Github Project? Try to compare and see what is missing on your project.

[Ruchir Baronia]

Hi Gal Rom, I actually did see your Github project, but I wasn't able to find out what was wrong in my project. Here is my stack trace. For some reason, I am not able to simply recognize for the word "hello". :( I would really appreciate any help, you seem to be the speech recognition expert!

[Gal Rom]

@RuchirBaronia The only way I can help you is by seeing the Project. Feel Free to send it to [email protected] and I will try to see what's up there.

[Mina Dahesh]

@NikolayShmyrev. may i ask u answer this question - stackoverflow.com/q/37225500/4568864 - thanks

[Mina Dahesh]

how can i set KEYWORD from edit text that user set it?

[Gaurav Arora]

@HoanNguyen I am always getting Error in SpeechRecognitionListener. Error - 9. Can you help me resolving the same. Have debugged at least 20times, but nothing has helped me yet.

[Gaurav Arora]

@HoanNguyen Thanks for the same. I am in a strange issue this time, that when I don't anything for 5-10 secs, my and then say something for 11th sec or nth sec, it never listens to me. Why ? I mean after onReadyForSpeech is not followed by onBeginning in that case.

[Gaurav Arora]

If I say something in starting e.g when the service is just started, it listens to me well and I can process the results. I want to make it to listen endlessly or at least for some minutes. But it waits only for few seconds

[mio]

@gal-rom your github code works! would it be possible for you to code in your github such that it can record audio in background (even when the screen is off, or the app is killed)? Something like this: fabcirablog.weebly.com/blog/…

[treesoft]

@GalRom thx a lot! I found a bug though: In onDestroy of SpeechRecognizerManager you check wether mIsStreamSolo is false, but you need to check if it is true, for reestablishing volume-settings!