How to use OpenCV 4 in native C++ of Flutter (in 2021) (support Flutter 2.0)?

Question

I need to write some C++ code which uses OpenCV, and the Flutter code will call those C++ code.

There are tutorials about writing C++ with Flutter, but I cannot find any up-to-date and easy-to-deploy solution about working with OpenCV. How to do that?

Solution

Here is my solution.

Features

1. Works for both Android and iOS.
2. Use static linking instead of dynamic linking. (Thus code size is much smaller.)
3. Up-to-date at 2022.04.03 working with OpenCV 4.5.5 and tested on both Flutter 1.x and 2.10.4. (Since those APIs change rapidly and many articles are a little bit old.)

Getting Started

NOTE: If you already have an app, you can skip this section :) This section assumes that you have no code at all.

The sample code can be downloaded from here.

step 0: Ensure you have Flutter environment, and have followed the official "writing C++ with Flutter" tutorial.

NOTE: It is a must to follow the step of "On iOS, you need to tell Xcode to statically link the file: ...". Otherwise, at our last step iOS will complain the symbol cannot be found.

NOTE: Also, you need to change the "strip style" as mentioned here for iOS. Otherwise, you will see problems in iOS release build while in debug build everything works.

step 1: Write whatever code you like using OpenCV. For instance, I change ios/Classes/native_add.cpp to the following silly code, which is almost identical as in the official tutorial:

#include <stdint.h>
#include <opencv2/core.hpp>

extern "C" __attribute__((visibility("default"))) __attribute__((used))
int32_t native_add(int32_t x, int32_t y) {
    cv::Mat m = cv::Mat::zeros(x, y, CV_8UC3);
    return m.rows + m.cols;
}

Android

Step 0: Download the Android OpenCV sdk from the official website. Say I put it in /Users/tom/Others/OpenCVRelease/OpenCV-android-sdk in my desktop.

Step 1.1: Change the android/CMakeLists.txt to the content of this gist. NOTE: First change the OPENCV_BASE_DIR to your folder.

Of course, the lib/native_with_opencv.dart should change the .so file name to "libnative_with_opencv.so".

Remark: If you need more OpenCV features (such as imread), have a try using this gist.

Step 1.2: Change the android/build.gradle as following:

android {
    ...
    defaultConfig {
        ...
        // [[[CHANGE 1: Make minSdkVersion bigger]]]
        // see https://github.com/opencv/opencv/issues/14419
        minSdkVersion 21

        // [[[CHANGE 2: Add these flags and filters]]]
        externalNativeBuild {
            cmake {
                cppFlags "-frtti -fexceptions -std=c++11"
                abiFilters 'armeabi-v7a', 'arm64-v8a'
            }
        }
        ...
    }
    ...
}

Of course, the minSdkVersion in your actual project (native_with_opencv/example/android/app/build.gradle) should also change to 21.

Done! Compile and enjoy it (and go to the next section for iOS)! If you see 1 + 2 == 3, then everything is fine.

Bonus: If you build in release mode and look at the apk size, you will see our .so file is less than 1MB. Thus static linking and file size reduction does work :)

iOS

Step 0: In ios/native_with_opencv.podspec, add:

  s.static_framework = true
  s.dependency 'OpenCV', '~> 4.5'

Step 1: Run pod install under native_with_opencv/example/ios.

Step 2: Compile and enjoy!

Remark 0: It is a must to follow the step of "On iOS, you need to tell Xcode to statically link the file: ..." in the tut. Otherwise, at our last step iOS will complain the symbol cannot be found.

Remark 0b: May need to check (verify) the following settings in XCode (which seems to be automatically included when dragging some files into XCode but not sure). Otherwise, your final IPA file (can be generated by this) will contain your .cpp source file besides compiled code, thus the source code is leaked.

The setting: Go to "Build Phase" of the "Runner" Target. (1) Look at "Copy Bundle Resources", and verify that your .cpp file or folders are not there. (2) Look at "Compile Sources", and verify your .cpp files are there. (You may need to first add your file in "Compile Sources" before removing it in "Copy Bundle Resources".)

Remark 1: If you are using other .hpp headers and see strange errors, such as OpenCV should be built with C++, or include of a non-modular header inside framework module, then may try this:

Create xxx.modulemap file containing the following framework module the_name_of_your_module {}. Then change your podspec to use this modulemap s.module_map = 'xxx.modulemap'. Then run pod install again to refresh. Then compile and run and it should be OK.

My guess about this problem is that, Cocoapod generates an "umbrella header" (say vision_utils-umbrella.h), and your header is automatically included there. Thus, when compiling that header, things get broken. Thus, my method above tries to remove this umbrella header.

Remark 2: When you add or remove some c++ files, you may have to run pod install again (just like in step 1). Otherwise, you may see errors like "cannot find the symbol" (because Xcode does not look at your newly added C++ file).

Remark 3: If you see problems like Failed to lookup symbol (dlsym(RTLD_DEFAULT, your_function): symbol not found), have a look at this link. I seem to solve it by applying what is suggested in that comment (but not sure since also have tried other ways). If still not solved please reply to me.

---

(Optional) Explanations of how do the Android configuration work: (1) Originally, I just link the core, but there are hundreds of linking errors. Then I search and fix for each group of them. For instance, error: undefined reference to 'carotene_o4t::...' means I need to link with libtegra_hal, thus I add several lines. (2) Strangely, the tbb should be put after core, otherwise, it still does not link. (3) The abiFilters is needed, since tegra_hal does not support x86 (thus no .a file exists). (4) minSdkVersion needs to be raised up, otherwise fegetenv will not be found.

Comments

[Schnodderbalken]

Great tutorial, thank you! I have a problem, though: as long as I stick to your example, using cv::Mat m = cv::Mat::zeros(x, y, CV_8UC3), everything works fine. However, if I want to use imread() instead, I get errors like error: undefined reference to 'Imf_opencv:: ..... I already added imgcodecs and imgproc to target_link_libraries.

[ch271828n]

@Schnodderbalken Maybe you can create a question and link it here? I am not sure about your full situation given this short comment.

[ch271828n]

@Schnodderbalken I have edited my answer with my latest CMakeLists.txt (see "If you need more OpenCV features...").

[Schnodderbalken]

Thank you very much! Now, everything's working as expected.

[ch271828n]

@Schnodderbalken Happy to hear that :)

[ykonda]

just a note, if using imgproc, you want to place that BEFORE core in the MakeLists.txt for android. So the order would be imgproc then core then tbb.

[ch271828n]

@ykonda thanks. I have updated the gist: gist.github.com/fzyzcjy/621fb575e7ca57beed6a55576418efd8

[Alexis]

@ch271828n Tanks. Also, do you have any reference for integrating OpenCV for flutter desktop applications?

[ch271828n]

@AlexisToby I have not tried that. It should be similar to how you integrate any windows/linux/macos libs to flutter

[Alexis]

@ch271828n Yes, but it's really complex, or maybe because am a beginner in Flutter, I couldn't make it working.

[ch271828n]

@AlexisToby You may make it a separate stack overflow question. I personally do not have the experience yet.

[Alexis]

@ch271828n Okay. Thanks.