How to get raw preview data from Camera object at least 15 frames per second in Android?

Question

I need to obtain raw preview data from Camera object at least 15 frame per second, but I can only get a frame in 110 milliseconds which means I can get only 9 frames per second. I brief my code below.

Camera mCamera = Camera.open();
Camera.Parameters parameters = mCamera.getParameters();
parameters.setPreviewFrameRate(30);
parameters.setPreviewFpsRange(15000,30000);
mCamera.setParameters(parameters);
mCamera.addCallbackBuffer(new byte[dataBufferSize]);
//dataBufferSize stands for the byte size for a picture frame
mCamera.addCallbackBuffer(new byte[dataBufferSize]);
mCamera.addCallbackBuffer(new byte[dataBufferSize]);
mCamera.setPreviewDisplay(videoCaptureViewHolder);
//videoCaptureViewHolder is a SurfaceHolder object
mCamera.setPreviewCallbackWithBuffer(new Camera.PreviewCallback() {
  private long timestamp=0;
  public synchronized void onPreviewFrame(byte[] data, Camera camera) {
    Log.v("CameraTest","Time Gap = "+(System.currentTimeMillis()-timestamp));
    timestamp=System.currentTimeMillis();
    //do picture data process
    camera.addCallbackBuffer(data);
    return;
  }
}
mCamera.startPreview();

In the briefed code above, dataBufferSize and videoCaptureViewHolder is defined and calculated or assigned in other statements.

I run my code, I can see preview on the screen and I get the log below:

...
V/CameraTest( 5396): Time Gap = 105
V/CameraTest( 5396): Time Gap = 112
V/CameraTest( 5396): Time Gap = 113
V/CameraTest( 5396): Time Gap = 115
V/CameraTest( 5396): Time Gap = 116
V/CameraTest( 5396): Time Gap = 113
V/CameraTest( 5396): Time Gap = 115
...

This means onPreviewFrame(byte[] data, Camera camera) is called every 110 milliseconds so I can get no more than 9 frames per second. And no matter what preview frame rate I set by issue setPreviewFrameRate() and what preview Fps range I set by issue setPreviewFpsRange(), the log is the same.

Would some one give me some help on this problem? I need to obtain raw preview data from Camera object at least 15 frames per second. Thank you in advance.

I put my entire code below.

CameraTest.java

package test.cameratest;

import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import android.app.Activity;
import android.graphics.ImageFormat;
import android.hardware.Camera;
import android.hardware.Camera.ErrorCallback;
import android.hardware.Camera.Parameters;
import android.hardware.Camera.Size;
import android.os.Bundle;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.SurfaceHolder.Callback;

public class CameraTestActivity extends Activity {
    SurfaceView mVideoCaptureView;
    Camera mCamera;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
        mVideoCaptureView = (SurfaceView) findViewById(R.id.video_capture_surface);
        SurfaceHolder videoCaptureViewHolder = mVideoCaptureView.getHolder();
        videoCaptureViewHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
        videoCaptureViewHolder.addCallback(new Callback() {
            public void surfaceDestroyed(SurfaceHolder holder) {
            }

            public void surfaceCreated(SurfaceHolder holder) {
                startVideo();
            }

            public void surfaceChanged(SurfaceHolder holder, int format,
                    int width, int height) {
            }
        });
    }
    private void startVideo() {
        SurfaceHolder videoCaptureViewHolder = null;
        try {
            mCamera = Camera.open();
        } catch (RuntimeException e) {
            Log.e("CameraTest", "Camera Open filed");
            return;
        }
        mCamera.setErrorCallback(new ErrorCallback() {
            public void onError(int error, Camera camera) {
            }
        }); 
        Camera.Parameters parameters = mCamera.getParameters();
        parameters.setPreviewFrameRate(30);
        parameters.setPreviewFpsRange(15000,30000);
        List<int[]> supportedPreviewFps=parameters.getSupportedPreviewFpsRange();
        Iterator<int[]> supportedPreviewFpsIterator=supportedPreviewFps.iterator();
        while(supportedPreviewFpsIterator.hasNext()){
            int[] tmpRate=supportedPreviewFpsIterator.next();
            StringBuffer sb=new StringBuffer();
            sb.append("supportedPreviewRate: ");
            for(int i=tmpRate.length,j=0;j<i;j++){
                sb.append(tmpRate[j]+", ");
            }
            Log.v("CameraTest",sb.toString());
        }

        List<Size> supportedPreviewSizes=parameters.getSupportedPreviewSizes();
        Iterator<Size> supportedPreviewSizesIterator=supportedPreviewSizes.iterator();
        while(supportedPreviewSizesIterator.hasNext()){
            Size tmpSize=supportedPreviewSizesIterator.next();
            Log.v("CameraTest","supportedPreviewSize.width = "+tmpSize.width+"supportedPreviewSize.height = "+tmpSize.height);
        }

        mCamera.setParameters(parameters);
        if (null != mVideoCaptureView)
            videoCaptureViewHolder = mVideoCaptureView.getHolder();
        try {
            mCamera.setPreviewDisplay(videoCaptureViewHolder);
        } catch (Throwable t) {
        }
        Log.v("CameraTest","Camera PreviewFrameRate = "+mCamera.getParameters().getPreviewFrameRate());
        Size previewSize=mCamera.getParameters().getPreviewSize();
        int dataBufferSize=(int)(previewSize.height*previewSize.width*
                               (ImageFormat.getBitsPerPixel(mCamera.getParameters().getPreviewFormat())/8.0));
        mCamera.addCallbackBuffer(new byte[dataBufferSize]);
        mCamera.addCallbackBuffer(new byte[dataBufferSize]);
        mCamera.addCallbackBuffer(new byte[dataBufferSize]);
        mCamera.setPreviewCallbackWithBuffer(new Camera.PreviewCallback() {
            private long timestamp=0;
            public synchronized void onPreviewFrame(byte[] data, Camera camera) {
                Log.v("CameraTest","Time Gap = "+(System.currentTimeMillis()-timestamp));
                timestamp=System.currentTimeMillis();
                try{
                    camera.addCallbackBuffer(data);
                }catch (Exception e) {
                    Log.e("CameraTest", "addCallbackBuffer error");
                    return;
                }
                return;
            }
        });
        try {
            mCamera.startPreview();
        } catch (Throwable e) {
            mCamera.release();
            mCamera = null;
            return;
        }
    }
    private void stopVideo() {
        if(null==mCamera)
            return;
        try {
            mCamera.stopPreview();
            mCamera.setPreviewDisplay(null);
            mCamera.setPreviewCallbackWithBuffer(null);
            mCamera.release();
        } catch (IOException e) {
            e.printStackTrace();
            return;
        }
        mCamera = null;
    }
    public void finish(){
        stopVideo();
        super.finish();
    };
}

AndroidManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="test.cameratest"
      android:versionCode="1"
      android:versionName="1.0">
    <uses-sdk android:minSdkVersion="9" android:targetSdkVersion="10" android:maxSdkVersion="10"/>
    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.VIBRATE" />
    <uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
    <uses-permission android:name="android.permission.WAKE_LOCK" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.MOUNT_UNMOUNT_FILESYSTEMS"/>
    <uses-permission android:name="android.permission.READ_CONTACTS"/>
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
    <uses-permission android:name="android.permission.PROCESS_OUTGOING_CALLS"/>
    <uses-permission android:name="android.permission.CALL_PHONE"/>
    <uses-permission android:name="android.permission.BOOT_COMPLETED"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.WRITE_SETTINGS" />    
    <application android:icon="@drawable/icon" android:label="@string/app_name">
        <activity android:name=".CameraTestActivity"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>
</manifest>

Solution 1

I'm affraid, you can not. Preview framerate setting is hint for camera appplication ( which runs in separate process) - and it is free to accept or silently ignore it. It is also not related with preview frame retrieval

When you request preview frame, you just say external application that you would like to have it. Buffer for it is allocated in camera application and then passed to your activity via mmaped memory segment - this takes time.

You may get desired performance on some devices, but not necessarily on one you are playing with.

If you need defined frame rate, you will have to capture video and then parse / decompress resulting binary stream.

Solution 2

Me experience with the camera stuff has been fiddly and hardware dependent. So try running it on other hardware sometime if you can.

Also might be worth trying some more camera settings.

• Checkout setRecordingHint (boolean hint).
• Could even try FOCUS_MODE.

Thanks for including a code sample btw.

Solution 3

This should not be a problem. My androangelo (it's in the market) app get's up to 30 frames per second (at least I implemented a rate-brake to slow it down).

Please check carefully, whether Your log is filled with garbage-collector statements. This is the case if too few buffers are added. This had be the trick for me. At least I came up to add 20! buffers to the camera.

Then the processing of each frame should take place in a separate thread. While an image is in the thread for processing, the callback should skip the current frame.

Solution 4

In my understanding, Android does not allow user to a set a fixed framerate, nor guarantee the value of fps that you specify will be respected is due to the frame exposure time which is set by the camera hardware or firmware. The frame rate you observe may be a function of lighting conditions. For example, a certain phone may give you a 30fps preview rate in a day light but only 7 fps if you are filming in a low light condition.

Solution 5

One thing that seems to increase the fluidity of the preview, if not the actual FPS necessarily, is setting the previewFormat to YV12 if supported. It's fewer bytes to copy, 16-byte aligned, and possibly optimized in other ways:

    // PREVIEW FORMATS
    List<Integer> supportedPreviewFormats = parameters.getSupportedPreviewFormats();
    Iterator<Integer> supportedPreviewFormatsIterator = supportedPreviewFormats.iterator();
    while(supportedPreviewFormatsIterator.hasNext()){
        Integer previewFormat =supportedPreviewFormatsIterator.next();
        // 16 ~ NV16 ~ YCbCr
        // 17 ~ NV21 ~ YCbCr ~ DEFAULT
        // 4  ~ RGB_565
        // 256~ JPEG
        // 20 ~ YUY2 ~ YcbCr ...
        // 842094169 ~ YV12 ~ 4:2:0 YCrCb comprised of WXH Y plane, W/2xH/2 Cr & Cb. see documentation
        Log.v("CameraTest","Supported preview format:"+previewFormat);
        if (previewFormat == ImageFormat.YV12) {
            parameters.setPreviewFormat(previewFormat);
            Log.v("CameraTest","SETTING FANCY YV12 FORMAT");                
        }
    }

http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12 describes the format. This plus a few spare buffers gives me "Time Gaps" of as low as 80...which is still not "good enough", but ... better? (actually I've got one at 69...but really, they're more around 90 on average). Not sure how much logging is slowing things down?

Setting the previewSize to 320x240 (versus 1280x720) gets things down to the 50-70msec range...so maybe that's what you need to do? Admittedly, that little data may be a lot less useful.

// all tested on Nexus4

View more solutions

Comments

[EricOops]

In fact I get the specification of setPreviewFpsRange(int min, int max) in <http:///developer.android.com/reference/android/hardware/Ca‌​mera.Parameters.html‌​#setPreviewFpsRange(‌​int, int)> . It said this API can "Sets the maximum and maximum preview fps. This controls the rate of preview frames received in Camera.PreviewCallback." So I think maybe by using this API I can get more than 15 frames per second. Does my assumption feasible? By the way, the supported preview Fps range of my device is (1000,15000) and (5000,30000) by issuing parameters.getSupportedPreviewFpsRange() API.

[Konstantin Pribluda]

Well, it also says that actual range fluctuates between those values. ( so if you set 1000-15000 you wil get something in between, not especially stable) I would not rely on this, as sttering camera behavour requires a lot of darkest voodoo

[EricOops]

I also have a doubt about the effect of this API. But no matter I setPreviewFpsRange(1000,1000) or setPreviewFpsRange(20000,30000), I get the same log. So I am quite confused. Does this API totally not work on my device? Or does this API not works as the way I assumed. Or Did I miss some factor or some statements? So I try to fix this problem by sticking this API at first. I will try your advice if I can not get through. Thank you very much. :)

[Konstantin Pribluda]

I use obly single snapshots (OCR processing takes about 1 second). AFAIK, only advertised ranges can be used - but I also expirienced that not all preview resolutions are actually usable, despite being advertised (that's how I discovered that mmaped buffer is used to pass frames - got a nice segv). I think you will have to stick with this API - there is no aletrnative.

[EricOops]

In fact I just record system time in the onPreviewFrame to calculate the time gap between every callback. It finishes very quickly. So I think no frame will be skipped. And I tried your advice. I added 20, 30 and even more buffers, but the result remained the same. In the log, there is a few garbage-collector statements, and I think it's not related to this part.

[DonSteep]

If you require a way to block code from being executed, be aware that this code does not achieve fool-proof locking.

[Alex Cohn]

For all it's worth, YV12 is not more efficient than NV21, and even if it is supported by Camera, there are no methods to convert it to Jpeg or Bitmap.

[Alex Cohn]

This won't work, because the callbacks all arrive on the same thread (and unless you prepare a separate looper for that, it will be the main looper, congesting the UI thread). For this pattern to work, you should offload the processing to a separate thread.