How to add OpenCV in Flutter Plugin

Question

I am working on a flutter project where I have to detect edges of the objects. I have searched for existing plugins but couldn't find any. I am planning to use OpenCV for edge detection.

I have created a flutter plugin from Android Studio. This is the plugin structure.

I have downloaded the OpenCV for Android from here - opencv-3.4.3-android-sdk and extracted.

I have imported the java folder from OpenCV SDK as the module to the Android Project of the flutter plugin.

This is how my android project (flutter plugin) build.gradle looks like

group 'com.xxxx.xxxxx'
version '1.0-SNAPSHOT'

buildscript {
    ext.kotlin_version = '1.2.30'
    repositories {
        google()
        jcenter()
    }

    dependencies {
        classpath 'com.android.tools.build:gradle:3.1.2'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
    }
}

rootProject.allprojects {
    repositories {
        google()
        jcenter()
    }
}

apply plugin: 'com.android.library'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'

android {
    compileSdkVersion 27

    sourceSets {
        main.java.srcDirs += 'src/main/kotlin'
    }
    defaultConfig {
        minSdkVersion 16
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    lintOptions {
        disable 'InvalidPackage'
    }
}

dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib-jre7:$kotlin_version"
    implementation 'com.android.support:appcompat-v7:27.1.1'
    implementation 'io.reactivex.rxjava2:rxandroid:2.1.0'

    implementation project(':openCVLibrary343')
    compileOnly files('templibs/flutter.jar')
}

This is how my opencv build.gradle looks like

apply plugin: 'com.android.library'

android {
    compileSdkVersion 27
    buildToolsVersion "28.0.3"

    defaultConfig {
        minSdkVersion 8
        targetSdkVersion 21
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.txt'
        }
    }
}

This is how the settings.gradle of the Android project(flutter plugin) looks like

rootProject.name = 'flutter_plugin'
include ':openCVLibrary343'

When I am trying to run the example app that is created by the flutter plugin template I am getting below error.

Launching lib\main.dart on Nexus 6P in debug mode... Initializing gradle... Resolving dependencies... * Error running Gradle: Exit code 1 from: C:\Users\xxxxx\Documents\flutter_plugin\example\android\gradlew.bat app:properties:

FAILURE: Build failed with an exception.

• Where: Build file 'C:\Users\xxxx\Documents\flutter_plugin\android\build.gradle' line: 48

• What went wrong: A problem occurred evaluating project ':flutter_plugin'.

  Project with path ':openCVLibrary343' could not be found in project ':flutter_plugin'.

• Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

• Get more help at https://help.gradle.org

BUILD FAILED in 1s

Finished with error: Please review your Gradle project setup in the android/ folder.

Solution

For a project I'm working on, I've managed to create a flutter plugin which relies on OpenCV in order to process images coming from the camera. Please be aware that when I say "from the camera", I not mean real time frames from video capture, but pictures previously saved on device memory.

First of all, I followed the Flutter docs regarding creation of plugins here

When flutter has created the plugin folder and files, I've ended up with a structure similar to this (the "opencv" folder at first level is the root of plugin):

Notice the opencv-341.jar file which I added and contains the Java methods to control the native libraries, the OpenCVPlugin.java and opencv.dart which are the entry point to access Java methods from Flutter, and were created from Flutter.

The files name of .java and .dart files could be different and reflect the name you passed as argument during the plugin creation phase above.

The .jar file can be found in OpenCV package for JAVA as precompiled library. In same package you will find the native libraries needed at the end of this little guise.

The build.gradle file in [plugin_root]/android should reference the jar file like this

dependencies {
    implementation files('opencv/opencv-341.jar')
}

The OpencvPlugin.java should host the methods invoked from dart. In my case, to better manage different methods, I've created a code like this:

if (call.method.equals("processReceiptImage")) {
    if (!call.hasArgument("input")) {
        result.error("NOTFOUND", "File not found", null);
        return;
    }
    try {
        String input = call.argument("input");
        int maxWidth = (int) call.argument("maxWidth");
        int quality = (int) call.argument("quality");
        String output = call.argument("output");
        result.success(ProcessReceiptImage.run(input, maxWidth, quality, output));
        return;
    } catch (Exception e) {
        e.printStackTrace();
        result.error("UNKNOWN", e.getMessage(), null);
        return;
    }
}

The static method ProcessReceiptImage is contained in the relative file ProcessReceiptImage.java:

package com.yourplugin.opencv;

import org.opencv.core.Mat;
import org.opencv.core.MatOfInt;
import org.opencv.core.Size;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

import java.util.ArrayList;


public class ProcessReceiptImage {
    static String run(String input, int maxWidth, int quality, String output) throws Exception {
        Mat src = Imgcodecs.imread(input, Imgcodecs.CV_LOAD_IMAGE_GRAYSCALE);
        Size size = src.size();

        int height = (int) (maxWidth * size.height / size.width);
        Imgproc.resize(src, src, new Size(maxWidth, height));

        ArrayList<Integer> parameters;
        parameters = new ArrayList();

        parameters.add(Imgcodecs.CV_IMWRITE_JPEG_QUALITY);
        parameters.add(quality);

        MatOfInt par = new MatOfInt();
        par.fromList(parameters);
        Imgcodecs.imwrite(output, src, par);
        return output;
    }
}

Since this point, if you run your app, it could fail due to the lack of library opencv_java3. In order to resolve this part, the only way I founded by now is to copy the native libraries into your final project like this (this is the root of your app, not the plugin):

I've included the final .so native library for each platform I want to target, so no matter what platform has the user, the code always can pick up the corret .so and correctly load OpenCV.

TIP: Since including all .so files can lead to a large APK, I suggest you to configure your build.gradle in order to split the package for abis, this way you will have a smaller apk which will be automatically served through Play Store.

Comments

[Sawan Kumar Bundelkhandi]

I have followed your steps and I have created this plugin. github.com/sawankumarbundelkhandi/edge_detection Thanks

[Nick]

Hi Sawan, I can't find your plugin in pub.dev repositories. When you will publish the plugin?

[Sawan Kumar Bundelkhandi]

You can install that plugin directly from GitHub. You have to add in your pubspec.yam file (in dependencies part): edge_detection: git: https://github.com/sawankumarbundelkhandi/edge_detection.git

[DCodes]

@SawanKumarBundelkhandi can you please provide me a way to detect objects from gallery images?