Use Protobuf 3 In Flutter's C++ Code - How To Compile And Link?

Question

Protobuf is a very widely used library, so I want to use it in Flutter's C++ code to serialize and deserialize my data. However, I find that compiling and linking the protobuf library is not a trivial task... I failed many times before finding out the correct solution, so I want to share it here using Q&A style.

Solution

Preparation

Firstly, of course, you need to generate your protobuf files. This is out the scope of this tutorial, so you can look at the official guide. Suppose you already have hello.pb.h and hello.pb.cc generated.

iOS

Add the following to your xxx.podspec, if you are developing a package/plugin that uses C++ code which needs protobuf. If it is your may code that needs protobuf, things are also similar.

Pod::Spec.new do |s|
  ...normal things...

  s.pod_target_xcconfig = {
    'HEADER_SEARCH_PATHS' => '$(SRCROOT)/Protobuf-C++/src',
  }
  
  s.dependency 'Protobuf-C++', '~> 3.18.0'
end

In addition, you need to add the following in your main project's Podfile. For example, if you are developing a package that is used by 10 projects, each of them needs to add the following section.

post_install do |installer|
  ...normal things...
  
  installer.pods_project.targets.each do |target|
    # print "target=", target, "\n"
    if target.name == "Protobuf-C++"
      target.build_configurations.each do |config|
        config.build_settings['HEADER_SEARCH_PATHS'] = '$(SRCROOT)/Protobuf-C++/src'
      end
    end
  end
end

Then clean and build the project and enjoy!

Remark: Why I need to hack the HEADER_SEARCH_PATHS: Originally it errors and says that protobuf headers cannot be found. This solution suggests that we add the paths manually by hand after each pod install. Moreover, this improved solution suggests that we can automate the process by adding a few lines of code. That is what we did above.

The alternative way

This link works pretty well basically. Although it is under the SDK tutorial of Cardboard, this webpage talks about almost nothing about Cardboard-specific things but only protobuf. Thus, feel free to follow it. After all steps except the last step ("copying files"), you will get: include/google/* and lib/libprotobuf-lite.a.

Before getting started, do not forget make distclean to clean everything (if you have already done something).

Remark 1: If you see errors when linking, saying "Undefined symbols" and "ld: warning: ignoring file", you may follow this link to solve it.

Remark 2: That tutorial only creates libprotobuf-lite.a. If you need libprotobuf.a, it can be done by analogy: lipo $build_dir/arch/arm64/lib/libprotobuf.a $build_dir/arch/armv7/lib/libprotobuf.a -create -output $build_dir/lib/libprotobuf.a.

Then, you can put the .a and the headers into your ios xcode project. I have not tried this yet, because I use the approach above.

Android

Get lib and headers

It is a little bit more complex than iOS.

Before getting started, do not forget make distclean to clean everything (if you have already done something).

Firstly, follow the same link as the approach to generating your own .a and headers in iOS to do git clone, git checkout, git submodule and ./autogen.sh, but stop there and do not continue.

Next, you need to follow this answer to hack the code a little bit - otherwise the compilation will fail. Please refer to the link to do a patch using ltmain.sh.patch. (As for the fuse-ld flag I already do it, so no need for you to do anything.)

Now comes the configure and compile, which goes as follows.

export NDKDIR=/Users/tom/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64
export SYSROOT=$NDKDIR/sysroot

./configure \
--prefix=/Users/tom/RefCode/libprotobuf/android \
--host=arm-linux-androideabi \
--with-sysroot="${SYSROOT}" \
--enable-shared \
--enable-cross-compile \
--with-protoc=protoc \
"CC=$NDKDIR/bin/armv7a-linux-androideabi26-clang" \
CFLAGS="-fPIC -fuse-ld=bfd -march=armv7-a -D__ANDROID_API__=26" \
"CXX=$NDKDIR/bin/armv7a-linux-androideabi26-clang++" \
CXXFLAGS="-fPIC -fuse-ld=bfd -frtti -fexceptions -march=armv7-a -D__ANDROID_API__=26" \
"RANLIB=$NDKDIR/bin/x86_64-linux-android-ranlib" \
"C_COMPILER_RANLIB=$NDKDIR/bin/x86_64-linux-android-ranlib" \
"CXX_COMPILER_RANLIB=$NDKDIR/bin/x86_64-linux-android-ranlib" \
"AR=$NDKDIR/bin/x86_64-linux-android-ar" \
LIBS="-llog -lz -lc++_static";
make -j8 && make install

Remark: The commands above is similar but not equal to this post, because the NDK standalone toolchain used in that post is deprecated already. Of course, you may need to change your versions, such as the path to your own ndk, your ndk version, etc.

Remark: You may also need to know the naming conventions used above. For example, why sometimes it is llvm, why sometimes it is darwin, and why armv7a. They are described in detail in the official guide.

Remark: this link is also useful when you face problem of libprotobuf.a: no archive symbol table (run ranlib) when compiling your android app using Gradle.

Remark: Without -fPIC, errors like requires unsupported dynamic reloc R_ARM_REL32; recompile with -fPIC will occur. So I followed this link to address it.

Remark: If you see errors such as error: undefined reference to 'stderr', you may need to make your minSdkVersion bigger, such as changing to 23. ref

Remark: After changing things, multiple cleaning may be needed. For example, gradle sync, gradle clean, make distclean, etc.

Similar to the guide, the next step is make -j8 and then make install. Then you are done and will see some libs like libprotobuf-lite.a as well as include headers.

Use it in your project

Add the following to CMakeLists.txt:

set(OPENCV_BASE_DIR "/some/absolute/path/to/your/base/dir/that/has/built/just/now")

message("PROTOBUF_BASE_DIR: ${PROTOBUF_BASE_DIR}")

set(PROTOBUF_INCLUDE_DIR "${PROTOBUF_BASE_DIR}/include/")
set(PROTOBUF_STATIC_LIB_DIR "${PROTOBUF_BASE_DIR}/lib")

if (NOT EXISTS ${PROTOBUF_INCLUDE_DIR})
    message(FATAL_ERROR "PROTOBUF_INCLUDE_DIR=${PROTOBUF_INCLUDE_DIR} does not exist - have you provided the correct PROTOBUF_BASE_DIR=${PROTOBUF_BASE_DIR}")
endif ()

include_directories(${PROTOBUF_INCLUDE_DIR})

add_library(protobuf-lite STATIC IMPORTED)
set_target_properties(protobuf-lite PROPERTIES IMPORTED_LOCATION ${PROTOBUF_STATIC_LIB_DIR}/libprotobuf-lite.a)

add_library(protobuf STATIC IMPORTED)
set_target_properties(protobuf PROPERTIES IMPORTED_LOCATION ${PROTOBUF_STATIC_LIB_DIR}/libprotobuf.a)

target_link_libraries(yourapp
        ...others...

        protobuf
        )

Since I only build for armv7 instead of armv8 in the code above, we can add an abi filter as follows. Or, alternatively, you can also build for armv8 and merge them into a fat lib just like what we have done in ios.

build.gradle:

android {
    defaultConfig {
        externalNativeBuild {
            cmake {
                abiFilters 'armeabi-v7a'
            }
        }
    }
}

Bonus: MacOS

Get lib and headers

What if you want to also use your C++ code (with protobuf) on your mac computer?

Before getting started, do not forget make distclean to clean everything (if you have already done something).

Similar to this link, you only need to change your ./configure to:

./configure CC=clang CXX="clang++ -std=c++11 -stdlib=libc++" CXXFLAGS="-O3" --disable-shared

and the other things are remained almost the same as the guide, such as downloading, configuring, make and finally sudo make install (notice you need sudo here).

You will see headers in your system's include directory and libs in your system's lib folder.

Use it in your project

If you are using CMake, it is mostly standard. Firstly, add path/to/your/hello.pb.cc into your add_executable. Secondly, add /usr/local/lib/libprotobuf.a to your target_link_libraries. Full example:

add_executable(app
        ./main.cpp
        path/to/your/hello.pb.cc
        ... others ...
        )
        
target_link_libraries(app
        /usr/local/lib/libprotobuf.a
        ... others ...
        )

Remark: libprotobuf-lite.a sometimes does not suffice, so you need to link with the big lib instead of the lite lib.