OpenCV Android Background Subtraction

Question

I am working on a robotics project using an Android phone as the main processor and the camera to detect movement. I got the Android binary package from OpenCV and got it correctly installed. I can capture images using the OpenCV native camera and display them to the screen. I'm having problems using the background subtraction class, though. I can make a new BackgroundSubtractorMOG object in the constructor, but when I attempt to run the code below, it force quits I get the error "Only 1- and 3-channel 8-bit images are supported in BackgroundSubtractorMOG" from the native code. I tried changing Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA to Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGB, and then it doesn't force quit, but all I get is a black screen. I'm pretty sure bmp is still null with FRAME_RGB, because the screen stays black, and the fps counter I was drawing right after the bitmap (removed from the code posted below for clarity and as a troubleshooting step) doesn't show up.

I took a look at the OpenCV C++ code for this function (line 388 here), and the image type error occurs if the image type isn't CV_8UC1 or CV_8UC3, so I tried using the java CvType.CV_8UC3 instead of Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA in capture.retrieve(), but it force closed and I got a "Output frame format is not supported" error.

I'm guessing I've just got a type conversion problem, but I can't figure out for the life of me where OpenCV's Android-specific image types fit with their regular image types that are documented. Any help would be appreciated.

The variables:

private SurfaceHolder mHolder;
private VideoCapture mCamera;
private Mat mRgba;
private Mat mFGMask;
private BackgroundSubtractorMOG mBGSub;

My SurfaceView's run() function:

public void run() {    
    Bitmap bmp = null;

    synchronized (this) {
        if (mCamera == null)
            break;

        if (!mCamera.grab()) {
            Log.e(TAG, "mCamera.grab() failed");
            break;
        }

        processFrame(mCamera);
        bmp = Bitmap.createBitmap(mFGMask.cols(), mFGMask.rows(), Bitmap.Config.ARGB_8888);
        Utils.matToBitmap(mFGMask, bmp);
    }

    if (bmp != null) {
        Canvas canvas = mHolder.lockCanvas();
        if (canvas != null) {
            canvas.drawBitmap(bmp, (canvas.getWidth() - bmp.getWidth()) / 2, (canvas.getHeight() - bmp.getHeight()) / 2, null);
            mHolder.unlockCanvasAndPost(canvas);
        }
        bmp.recycle();
    }
}

The processFrame() function referenced in run():

protected void processFrame(VideoCapture capture) {
    capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA);
    mBGSub.apply(mRgba, mFGMask);
}

Edit:

The solution that ended up working:

protected void processFrame(VideoCapture capture) {
    capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGB);
    //GREY_FRAME also works and exhibits better performance
    //capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_GREY_FRAME);
    mBGSub.apply(mRgba, mFGMask, 0.1);
    Imgproc.cvtColor(mFGMask, mRgba, Imgproc.COLOR_GRAY2BGRA, 4);
}

Solution 1

Have you tried using cvtColor with CV_RGB2RGBA and CV_RGBA2RGB. So, maybe try converting frame RGBA to RGB, then do background subtraction. Something like this:

protected void processFrame(VideoCapture capture) {
    capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA);
    Mat rgb;
    Imgproc.cvtColor(mRgba, rgb, Imgproc.COLOR_RGBA2RGB);
    mBGSub.apply(rgb, mFGMask);
}

EDIT : You might check out the OpenCV unit-test for BackgroundSubtractorMOG located here. However, the test has fail("Not yet implemented"); in the main test case.

I'm not sure if that means the test isn't complete, or the support for BackgroundSubtractorMOG is not implemented. You might try running the code contained in this unit-test to see if it actually works.

Also, the C++ sample segment_objects.cpp might be helpful as a usage example.

Hope that helps! :)

Solution 2

Thanks you guys so much! And for future viewers who come to this page, you might have to tweak this knowledge to get things working. In SDK v2.4.4 I applied this in the onCameraFrame method. Recall that the method takes in an input frame from the camera. You use the input and return the frame that is to be displayed on the screen of your android device. Here's an example:

//Assume appropriate imports    
private BackgroundSubtractorMOG sub = new BackgroundSubtractorMOG(3, 4, 0.8);
private Mat mGray = new Mat();
private Mat mRgb = new Mat();
private Mat mFGMask = new Mat();

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    mGray = inputFrame.gray(); //I chose the gray frame because it should require less resources to process
    Imgproc.cvtColor(mGray, mRgb, Imgproc.COLOR_GRAY2RGB); //the apply function will throw the above error if you don't feed it an RGB image
    sub.apply(mRgb, mFGMask, learningRate); //apply() exports a gray image by definition

    return mFGMask;
}

To get across my point about the gray image that comes out of apply(), if you wanted to do a RGBA version, you might have to use a cvtcolor after the call to apply():

private Mat mRgba = new Mat();

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    mRgba = inputFrame.rgba();
    Imgproc.cvtColor(mRgba, mRgb, Imgproc.COLOR_RGBA2RGB); //the apply function will throw the above error if you don't feed it an RGB image
    sub.apply(mRgb, mFGMask, learningRate); //apply() exports a gray image by definition
    Imgproc.cvtColor(mFGMask, mRgba, Imgproc.COLOR_GRAY2RGBA);

    return mRgba;
}

Comments

[RedPeasant]

This gets me closer. The exact syntax for the java-wrapped version is: Imgproc.cvtColor(mRgba, rgb, Imgproc.COLOR_RGBA2RGB); for the first convert, and the second one isn't needed (I'm not doing anything with the raw image after the subtraction). Now, I get a black screen (no force close). If I change the bitmap format to Bitmap.Config.RGB_565, I get my FPS counter back, so I know it's running, but it's still a black screen. Thanks for getting me a little farther along, though.

[mevatron]

@RedPeasant Thanks for the info on the correct syntax. I've update my answer to include that. Is the camera moving and the screen is still black?

[mevatron]

@RedPeasant I have updated my answer to include more information on the BackgroundSubtractorMOG.

[RedPeasant]

Got it working. See the original question for the solution. I'm accepting your answer because you got me most of the way there. Thanks for your help.

[austin]

I am unsure on how to select values for the BackgroundSubtractorMOG constructor. I saw those on another thread. Make sure to play with the values to better fit your application.

[karanatwal.github.io]

Please have a look at this question @mevatron stackoverflow.com/q/43864870/5996106

[most venerable sir]

Thank you so much. My school project is saved.