Flutter Integration Tests with Travis CI

Question

1. The Summary of the Problem

I would like a Travis CI setup that would let me run flutter driver tests inside an Android and an iOS environments. For this to happen, I expect I somehow have to install Flutter, Android and iOS in different environments.

2. What I Have So Far

Most of the posts I've been able to find on this topic are terribly outdated or feature incredibly complicated setups. Some of those that keep on appearing in my searches are:

• Test Flutter apps on Travis, by Yegor Jbanov. This one covers unit and widget testing (flutter test), but not integration tests.

  • It's from early 2017 and Travis CI has maybe simplified its API, because I've managed to make it work with only this:

    language: dart
    
    dart:
      - stable
    
    dart_task:
      - dartfmt
    
    install:
      - git clone https://github.com/flutter/flutter.git -b stable
    
    script:
      - ./flutter/bin/flutter doctor
      - ./flutter/bin/flutter test
    

• One resource that I've found very useful is the .travis.yml in the Flutter samples repo. The setup there seems very complicated to me though.
• The closest I could get to what I wanted is similar to Maurice McCabe's Flutter unit, widget and integration testing with IOS and Android emulators on Travis-CI.
  • Again, this seems overcomplicated and outdated.

3. The Sketch of What I Have in Mind

The script and install steps in the example I mentioned previously could be replaced by jobs with stages. In this way, each stage would represent one sort of step. Unit and Widget stages in one, integration tests on Android and iOS in two others, which is similar to what Maurice McCabe and Flutter samples show. For example:

jobs:
  include:
    - stage: Flutter Test
      language: dart
      os: linux
      install: git clone $FLUTTER_GITHUB -b stable
      before_script:
        - ./flutter/bin/flutter doctor
      script: 
        - ./flutter/bin/flutter test
    - stage: Integration Test on Android
      os: linux
      dist: trusty
      language: android
      android: # the things here are what probably needs to be fixed
        components:
          - build-tools-28.0.3
          - android-28
      install: git clone $FLUTTER_GITHUB -b stable
      before_script:
        - ./flutter/bin/flutter doctor
      script: 
        - ./flutter/bin/flutter drive --target=test_driver/app.dart

If I could create a stage for the dartfmt task that would also be nice in terms of organization.

Solution 1

1. The Overview

Kudos to @MirceaMatei and Maurice McCabe for helping out in this.

1. I haven't yet been able to make Android Integration Tests work (the last stage of the code below), but at least iOS is working.
   • Android is much harder to get right because of different versions and licenses, something Apple does much better.
2. The code you will find below is part of a custom mono repo setup I'm working on.
   • The folder structure consists of an app and a packages folders.
3. I haven't yet segmented code coverage by those two folders either, so this setup is currently overwriting code coverage from one folder to another.
4. Using a top level install section makes Travis repeat that setup for every stage, so that avoids having to do that repeatedly for every stage.

I'm sharing my current — albeit incomplete — setup below, but continuous improvement will happen through this Github Gist, which is a much better way of interacting with the community to improve code than StackOverflow — it's not its purpose.

All in all, I think Travis CI is really dropping the ball when it comes to Flutter — and maybe Android in general. An easier alternative that has been hyped by many practitioners I've come into contact recently is Codemagic, which also offers code signing and automated deployment to the iOS and Android app stores.

2. Useful Resources

I didn't really like Travis documentation for the integration tests. People creating pure Android apps filed similar issues.

Anyway, here are some useful resources I've found during my searches:

1. Issue Comment on "Error: Target id is not valid. Use 'android list targets' to get the target ids."
2. “Invalid --abi armeabi-v7a for the selected target” with Google APIs
3. Is there a way to start android emulator in Travis CI build?
4. Travis-CI Android 28 licenses have not been accepted
5. Flutter Samples Travis Setup
6. @MirceaMatei's Travis Setup Recommendation
7. Travis Building Android Projects Documentation
8. Maurice McCabe's Unit and Integration Tests with Travis Article on Medium

3. The Code

Please, do not post code improvement suggestions here, but in the the Github Gist. I'll be updating the yaml code below myself every time a useful iteration arrives.

language: dart

env:
  global:
    - DARTSDK=./flutter/bin/cache/dart-sdk/bin
    - DARTFMT=$DARTSDK/dartfmt
    - FLUTTER=./flutter/bin/flutter
    - FLUTTER_UP=../flutter/bin/flutter
    - FLUTTER_GITHUB=https://github.com/flutter/flutter.git
    - CODECOV=https://codecov.io/bash
    - PACKAGES=packages
    - APP=app
    - ANDROID_API=28

install:
  - git clone $FLUTTER_GITHUB -b stable --depth 1
  - $FLUTTER doctor
  - $FLUTTER pub get

jobs:
  include:
    - stage: Formatting
      script:
        - $DARTFMT -n $PACKAGES --set-exit-if-changed

    - stage: Packages Flutter Test
      script:
        - $FLUTTER test --coverage $PACKAGES
      after_success:
        - bash <(curl -s $CODECOV)

    - stage: App Flutter Test
      script:
        - cd $APP
        - $FLUTTER_UP test --coverage
      after_success:
        - bash <(curl -s $CODECOV)

    - stage: iOS Integration Tests
      os: osx
      osx_image: xcode11
      before_script:
        - open /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app
        - export HOMEBREW_NO_AUTO_UPDATE=1
        - brew install libimobiledevice
        - brew install ideviceinstaller
        - brew install ios-deploy
        - brew install cocoapods || echo 'ignore exit(1)'
        - brew link --overwrite cocoapods
      script:
        - cd $APP
        - $FLUTTER_UP driver --target=test_driver/app.dart77

    - stage: Android Integration Tests
      language: android
      dist: trusty
      android:
        components:
          - tools
          - platform-tools
          - build-tools-25.0.3
          - android-24
          - android-22
          - extra-google-google_play_services
          - extra-google-m2repository
          - extra-android-m2repository
          - sys-img-armeabi-v7a-android-22
      licenses:
        - 'android-sdk-preview-license-52d11cd2'
        - 'android-sdk-license-.+'
        - 'google-gdk-license-.+'
      before_install:
        - yes | sdkmanager "platforms;android-28"
        - echo no | android create avd --force -n test -t android-22 --abi armeabi-v7a -c 32M
        - emulator -avd test -no-audio -no-window &
        - android-wait-for-emulator
        - adb devices
        - adb shell input keyevent 82 &
      script:
        - $FLUTTER --version
        - $FLUTTER doctor
        - cd $APP
        - $FLUTTER_UP devices
        - $FLUTTER_UP driver --target=test_driver/app.dart

Solution 2

You can take a look at this project

• Sqlite Scaffolding Generator @ Dart Framework ORM M8

I think it has all the components you are looking for. It's a Dart package that has, in the solution, an example integration project for Flutter.

The CI pipeline is built for Travis CI (based mostly on Maurice McCabe's article). It's possible that the travis.yml file has all the sections you need.

Comments

[Philippe Fanaro]

Very nice. As a coincidence, I was working on this again since yesterday. I'm gonna try to incorporate your solution to my attempts and see if it works, then I'll share my complete solution so we can maybe both learn from each other. Btw, you didn't create a stage for Android, did you? McCabe's article has one, and maybe you manage to get it to work but didn't bother putting it there?

[Philippe Fanaro]

I'm gonna invest today and maybe tomorrow into making this work, but then I might change my CI/CD to use Codemagic. It's much faster, since it's optimized for Flutter, much simpler, and some other people I talked to about this use Codemagic already. Travis would be nice because it's the standard for programming, but they really dropped the ball when it comes to Flutter and Dart quite frankly.

[Mircea Matei]

Good job. Sharing is caring! You're right, for Android, I avoided including the integration section because of licensing twists.

[Philippe Fanaro]

In the setup I shared above, I did manage to get rid of license errors. The error I'm getting is related to EGL, and also Check failed: android_surface_. Could not create an OpenGL, Vulkan or Software surface to setup rendering.. The stage gets stuck after that.