OpenCV / Android BufferQueue Error: Surface Texture Has Been Abandoned

Question

New to Android and OpenCV. Been trying to to implement code from new book, Mastering OpenCV with Practical Computer Vision Projects. The app basically renders cartoonified images on camera preview using OpenCV. You can touch the screen to save the cartoonified image.

Source code from authors is located here.

I made a small modification (see below) to CartoonifierApp.java file so that I can statically load the cartoonifier library with the OpenCV Manager app (the original code threw an UnsatisfiedLinkError).

The problem I'm facing is that when I load the app onto my Galaxy Nexus (Android 4.1.1) I get an empty black screen. My LogCat says:

E/BufferQueue(4744): [unnamed-4744-0] setBufferCount: SurfaceTexture has been abandoned! E/Cartoonifier::SurfaceView(4744): startPreview() failed

I think it's a memory issue. I know the cpp code works because it runs fine by itself on my computer -- although rendering is slow on a relatively new laptop (Asus U46E).

I have no idea how to fix this. The only helpful advice I found was here. If I replace my setPreview method in CatoonifierVewBase.java

 if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB)
     mCamera.setPreviewTexture( new SurfaceTexture(10) );
 else
     mCamera.setPreviewDisplay(null);

with

 mCamera.setPreviewDisplay(mHolder);

then what happens is the camera works like normal and I can save cartoonified images by touching the screen. Note that this is not exactly the desired outcome as I want to continually cartoonify image on camera preview. The camera is only working because I'm not writing to surface view (at least that's my understanding). The answer in the previously mentioned site has some workaround but I have no idea what he is talking about.

BTW I've tried all the sample OpenCV4Android apps and they work just fine. Also I'm using OpenCV version 2.4.3. API level target is 15.

Full Logcat:

12-22 15:33:07.966: I/CartoonifierApp(5999): Instantiated new class com.Cartoonifier.CartoonifierApp 12-22 15:33:07.966: I/CartoonifierApp(5999): called onCreate 12-22 15:33:07.966: I/CartoonifierApp(5999): Trying to load OpenCV library 12-22 15:33:07.982: I/Cartoonifier::SurfaceView(5999): Instantiated new class com.Cartoonifier.CartoonifierView 12-22 15:33:07.990: I/CartoonifierApp(5999): onResume 12-22 15:33:07.990: I/Cartoonifier::SurfaceView(5999): openCamera 12-22 15:33:07.990: I/Cartoonifier::SurfaceView(5999): releaseCamera 12-22 15:33:08.099: D/OpenCVManager/Helper(5999): Service connection created 12-22 15:33:08.099: D/OpenCVManager/Helper(5999): Trying to get library path 12-22 15:33:08.138: D/OpenCVManager/Helper(5999): Trying to get library list 12-22 15:33:08.169: D/OpenCVManager/Helper(5999): Library list: "" 12-22 15:33:08.169: D/OpenCVManager/Helper(5999): First attempt to load libs 12-22 15:33:08.169: D/OpenCVManager/Helper(5999): Trying to init OpenCV libs 12-22 15:33:08.169: D/OpenCVManager/Helper(5999): Trying to load library /data/data/org.opencv.engine/lib/libopencv_java.so 12-22 15:33:08.169:

D/dalvikvm(5999): Trying to load lib /data/data/org.opencv.engine/lib/libopencv_java.so 0x41936a40 12-22

15:33:08.193: D/dalvikvm(5999): Added shared lib /data/data/org.opencv.engine/lib/libopencv_java.so 0x41936a40 12-22

15:33:08.193: D/OpenCVManager/Helper(5999): OpenCV libs init was ok!

12-22 15:33:08.193: D/OpenCVManager/Helper(5999): First attempt to load libs is OK 12-22 15:33:08.193: D/OpenCVManager/Helper(5999): Init finished with status 0 12-22 15:33:08.193: D/OpenCVManager/Helper(5999): Unbind from service 12-22 15:33:08.200: D/OpenCVManager/Helper(5999): Calling using callback 12-22 15:33:08.200: I/CartoonifierApp(5999): OpenCV loaded successfully

12-22 15:33:08.200: D/dalvikvm(5999): Trying to load lib /data/data/com.Cartoonifier/lib/libcartoonifier.so 0x41936a40 12-22

15:33:08.200: D/dalvikvm(5999): Added shared lib /data/data/com.Cartoonifier/lib/libcartoonifier.so 0x41936a40 12-22

15:33:08.200: D/dalvikvm(5999): No JNI_OnLoad found in /data/data/com.Cartoonifier/lib/libcartoonifier.so 0x41936a40, skipping init 12-22 15:33:08.200: D/OpenCVManager/Helper(5999): Service connection created 12-22 15:33:08.200: D/OpenCVManager/Helper(5999): Trying to get library path 12-22

15:33:08.232: D/OpenCVManager/Helper(5999): Trying to get library list 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): Library list: ""

12-22 15:33:08.271: D/OpenCVManager/Helper(5999): First attempt to load libs 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): Trying to

init OpenCV libs 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): Trying to load library /data/data/org.opencv.engine/lib/libopencv_java.so 12-22 15:33:08.271: D/dalvikvm(5999): Trying to load lib /data/data/org.opencv.engine/lib/libopencv_java.so 0x41936a40 12-22

15:33:08.271: D/dalvikvm(5999): Shared lib '/data/data/org.opencv.engine/lib/libopencv_java.so' already loaded in same CL 0x41936a40 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): OpenCV libs init was ok! 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): First attempt to load libs is OK

12-22 15:33:08.271: D/OpenCVManager/Helper(5999): Init finished with status 0 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): Unbind from service 12-22 15:33:08.271: D/OpenCVManager/Helper(5999): Calling using callback 12-22 15:33:08.271: I/CartoonifierApp(5999): OpenCV loaded successfully 12-22 15:33:08.279: D/dalvikvm(5999): Trying to load lib /data/data/com.Cartoonifier/lib/libcartoonifier.so 0x41936a40 12-22 15:33:08.279: D/dalvikvm(5999): Shared lib '/data/data/com.Cartoonifier/lib/libcartoonifier.so' already loaded in same CL 0x41936a40 12-22 15:33:08.302: I/Cartoonifier::SurfaceView(5999): surfaceCreated 12-22 15:33:08.302: I/Cartoonifier::SurfaceView(5999): surfaceChanged(). Window size: 1196x670 12-22 15:33:08.302: I/Cartoonifier::SurfaceView(5999): setupCamera(1196x670) 12-22 15:33:08.302: I/Cartoonifier::SurfaceView(5999): Starting processing thread 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 1920x1080 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 1280x720 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 960x720 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 800x480 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 720x576 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 720x480 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 768x576 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 640x480 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 320x240 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 352x288 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 240x160 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 176x144 12-22 15:33:08.310: I/Cartoonifier::SurfaceView(5999): Found Camera Resolution 128x96 12-22 15:33:08.318: I/Cartoonifier::SurfaceView(5999): Chosen Camera Preview Size: 1280x720 12-22 15:33:08.333: D/dalvikvm(5999): GC_FOR_ALLOC freed 131K, 2% free 10807K/11011K, paused 13ms, total 13ms

12-22 15:33:08.333: I/dalvikvm-heap(5999): Grow heap (frag case) to 11.902MB for 1382416-byte allocation

12-22 15:33:08.357: D/dalvikvm(5999): GC_CONCURRENT freed 1K, 3% free 12156K/12423K, paused 12ms+1ms, total 24ms

12-22 15:33:08.357: D/dalvikvm(5999): WAIT_FOR_CONCURRENT_GC blocked 11ms

12-22 15:33:08.365: D/dalvikvm(5999): GC_FOR_ALLOC freed 0K, 3% free 12156K/12423K, paused 9ms, total 9ms

12-22 15:33:08.372: I/dalvikvm-heap(5999): Grow heap (frag case) to 13.219MB for 1382416-byte allocation

12-22 15:33:08.388: D/dalvikvm(5999): GC_CONCURRENT freed 0K, 3% free 13506K/13831K, paused 11ms+1ms, total 21ms

12-22 15:33:08.388: D/dalvikvm(5999): WAIT_FOR_CONCURRENT_GC blocked 7ms

12-22 15:33:08.404: D/dalvikvm(5999): GC_FOR_ALLOC freed <1K, 3% free 13506K/13831K, paused 9ms, total 10ms

12-22 15:33:08.411: I/dalvikvm-heap(5999): Grow heap (frag case) to 16.735MB for 3686416-byte allocation

12-22 15:33:08.427: D/dalvikvm(5999): GC_CONCURRENT freed <1K, 3% free 17106K/17479K, paused 12ms+1ms, total 22ms

12-22 15:33:08.427: D/dalvikvm(5999): WAIT_FOR_CONCURRENT_GC blocked 10ms

12-22 15:33:08.443: D/dalvikvm(5999): GC_FOR_ALLOC freed <1K, 3% free 17106K/17479K, paused 10ms, total 10ms

12-22 15:33:08.450: I/dalvikvm-heap(5999): Grow heap (frag case) to 20.250MB for 3686416-byte allocation

12-22 15:33:08.466: D/dalvikvm(5999): GC_CONCURRENT freed 0K, 2% free 20706K/21127K, paused 12ms+2ms, total 22ms

12-22 15:33:08.466: D/dalvikvm(5999): WAIT_FOR_CONCURRENT_GC blocked 5ms

12-22 15:33:08.466: I/Cartoonifier::SurfaceView(5999): start preview

12-22 15:33:08.497: E/BufferQueue(5999): [unnamed-5999-0] setBufferCount: SurfaceTexture has been abandoned!

12-22 15:33:08.505: E/Cartoonifier::SurfaceView(5999): mCamera.startPreview() failed

Snippet from CartoonifierApp.java showing my modification

private BaseLoaderCallback  mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
            case LoaderCallbackInterface.SUCCESS:
            {
                Log.i(TAG, "OpenCV loaded successfully");

                // Load native library after(!) OpenCV initialization
                System.loadLibrary("cartoonifier");
            } break;
            default:
            {
                super.onManagerConnected(status);
            } break;
        }
    }
};

/** Called when the activity is first created. */
@Override
public void onCreate(Bundle savedInstanceState) {
    Log.i(TAG, "called onCreate");
    super.onCreate(savedInstanceState);

    Log.i(TAG, "Trying to load OpenCV library");
    if (!OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback))
    {
      Log.e(TAG, "Cannot connect to OpenCV Manager");
    }

    requestWindowFeature(Window.FEATURE_NO_TITLE);

    mView = new CartoonifierView(this);
    setContentView(mView);

    // Call our "onTouch()" callback function whenever the user touches the screen.
    mView.setOnTouchListener(this);
}


@Override
protected void onPause() {
    Log.i(TAG, "onPause");
    super.onPause();
    mView.releaseCamera();
}

@Override
public void onResume()
{
    super.onResume();
    Log.i(TAG, "onResume");
    if (!OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback))
    {
      Log.e(TAG, "Cannot connect to OpenCV Manager");
    }   

    if( !mView.openCamera() ) {
        AlertDialog ad = new AlertDialog.Builder(this).create();  
        ad.setCancelable(false); // This blocks the 'BACK' button  
        ad.setMessage("Fatal error: can't open camera!");  
        /*ad.setButton("OK", new DialogInterface.OnClickListener() {  
            public void onClick(DialogInterface dialog, int which) {  
                dialog.dismiss();                      
                finish();
            }  
        });  */
        ad.show();
    }
}

Solution

This issue was workarounded in OpenCV some time ago.

Not sure if it is application or OS bug. The problem is that call to Bitmap.createBitmap detaches SurfaceTexture object used for visualization.

The workaround was to modify the setupCamera method of the base View class and change

try {
    setPreview();
} catch (IOException e) {
    Log.e(TAG, "mCamera.setPreviewDisplay/setPreviewTexture fails: " + e);
}

/* Notify that the preview is about to be started and deliver preview size */
onPreviewStarted(params.getPreviewSize().width, params.getPreviewSize().height);

to

/* Notify that the preview is about to be started and deliver preview size */
onPreviewStarted(params.getPreviewSize().width, params.getPreviewSize().height);

try {
    setPreview();
} catch (IOException e) {
    Log.e(TAG, "mCamera.setPreviewDisplay/setPreviewTexture fails: " + e);
}

(the order of lines is changed)