How to see line number of failure line on flutter integration test?
Solution 1
There is no way you can find an error line from Console Output, the best way to debug your code for error is to add a print statement before every driver
action, this is what I have found when I tried to find any solutions regarding this issue, test will look like this just added snippet of one of my tests:
test('sign up test, log in button on sign up is working', () async {
print('Sign-up test started');
print('Waiting for join now button');
await driver.waitFor(loginpage.joinNowButtonFinder);
print('Tap for join now button');
await driver.tap(loginpage.joinNowButtonFinder);
print('Waiting for join now screen');
await driver.waitFor(joinpage.nameFieldFinder);
print('Tap on login button in join now screen');
await driver.scrollIntoView(joinpage.loginButtonFinder);
await driver.tap(joinpage.loginButtonFinder);
print('Tap on signup again');
await driver.waitFor(loginpage.signupButtonInLoginFinder);
await driver.tap(loginpage.signupButtonInLoginFinder);
}, timeout: Timeout(Duration(minutes: 2)));
Solution 2
You can attach your tests to the running flutter application. You need to set VM_SERVICE_URL environment variable and then run tests with a different command
In Windows you can create ps1 file with this content and run it after app starts on the device or in the vm:
$env:VM_SERVICE_URL = adb logcat -d | `
Select-String -Pattern "Observatory listening on (.*)" | `
select-object -Last 1 | foreach-object { $_.Matches.Groups[1].Value }
$port = $env:VM_SERVICE_URL | `
Select-String -Pattern ".*\d:(\d*)/" | `
select-object -Last 1 | `
foreach-object { $_.Matches.Groups[1].Value }
adb forward tcp:$port tcp:$port
flutter packages pub run test $args
In *nix I use:
export VM_SERVICE_URL=$(adb logcat -d | grep -o -E "Observatory listening on (.*)" | tail -1 | cut -d" " -f 4)
port=$(echo $VM_SERVICE_URL | cut -d: -f3 | cut -d/ -f1)
adb forward tcp:$port tcp:$port
flutter packages pub run test $1 --tags $2
Running tests in such way allows you to launch application inside the debugger with all necessary breakpoints and then run tests against that app so you can check all the failures right in the running app
Ajay Gautam
Updated on December 01, 2022Comments
-
Ajay Gautam over 1 year
EDIT: Found issue at flutter issue list: https://github.com/flutter/flutter/issues/65952
When there is a failure on
flutter drive
- integration tests, where do I look for - for the line number at which the failure occurred? I can't seem to find it in theflutter drive
output!Any help / pointers will be appreciated. Thanks in advance.
Command used:
flutter drive --target=test_driver/app.dart
Output:
Warning: You are using these overridden dependencies: ! pedantic 1.10.0-nullsafety Running "flutter pub get" in aip_app... 865ms Running Xcode build... └─Compiling, linking and signing... 3.4s Xcode build done. 10.1s 00:00 +0: Integration tests (setUpAll) VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:60646/0zFtd5Zu3Rs=/ VMServiceFlutterDriver: Isolate found with number: 3931090751816747 VMServiceFlutterDriver: Isolate is paused at start. VMServiceFlutterDriver: Attempting to resume isolate flutter: ===== STARTING APP ===== flutter: 2020-12-19 16:02:02.193821: MainApp INFO: ===== STARTING APP ===== VMServiceFlutterDriver: Connected to Flutter application. flutter: 2020-12-19 16:02:02.384157: Utils INFO: Supported Biometrics: [] flutter: 2020-12-19 16:02:02.394761: MainApp INFO: Build tag calculated: v1.0.1+5 flutter: 2020-12-19 16:02:02.469009: AppSettingsBloc INFO: Building App Data with brightness Brightness.dark, buildTag: v1.0.1+5 flutter: 2020-12-19 16:02:02.799421: MainApp INFO: AppNavigationObserver didPush from null to CupertinoPageRoute<dynamic>(RouteSettings("/", null), animation: AnimationController#66eb9(⏭ 1.000; paused; for CupertinoPageRoute<dynamic>(/))) flutter: 2020-12-19 16:02:03.075460: BackendProvider INFO: ====>>>> StateChange to: UserInfo: Anon: false, TCtmVckk7dMFqbCco6h9NhGBiA42, null, null, null, Instance of 'UserData' flutter: 2020-12-19 16:02:03.089937: SimpleBlocDelegate INFO: onTransition Transition { currentState: BackendInitialized, event: SignInAtStartupEvent, nextState: BackendActionInProgress } flutter: 2020-12-19 16:02:03.093207: BackendProvider INFO: Hello Anon. Loading... 00:02 +1 ~1: Integration tests cash games flow flutter: Handling request from Flutter driver: navigate_to_root flutter: 2020-12-19 16:02:04.279687: SimpleBlocDelegate INFO: onTransition Transition { currentState: BackendActionInProgress, event: SignInAtStartupEvent, nextState: BackendDataLoaded } flutter: 2020-12-19 16:02:04.492851: MainApp INFO: AppNavigationObserver didPush from CupertinoPageRoute<dynamic>(RouteSettings("/", null), animation: AnimationController#66eb9(⏭ 1.000; paused; for CupertinoPageRoute<dynamic>(/))) to ===||MainMenuScreen||=== [PageTransition<dynamic>(RouteSettings("/main-menu", null), animation: AnimationController#ff2aa(▶ 0.000; for PageTransition<dynamic>))] flutter: 2020-12-19 16:02:04.973825: MainApp INFO: AppNavigationObserver didPush from ===||MainMenuScreen||=== [PageTransition<dynamic>(RouteSettings("/main-menu", null), animation: AnimationController#ff2aa(⏭ 1.000; paused; for PageTransition<dynamic>))] to ===||CashGamesWelcomeScreen||=== [PageTransition<dynamic>(RouteSettings("CashGamesWelcomeScreen", null), animation: AnimationController#ed347(▶ 0.000; for PageTransition<dynamic>))] flutter: 2020-12-19 16:02:05.418700: MainApp INFO: AppNavigationObserver didPush from ===||CashGamesWelcomeScreen||=== [PageTransition<dynamic>(RouteSettings("CashGamesWelcomeScreen", null), animation: AnimationController#ed347(⏭ 1.000; paused; for PageTransition<dynamic>))] to ===||CashGamesNewGameScreen||=== [PageTransition<dynamic>(RouteSettings("CashGamesNewGameScreen", null), animation: AnimationController#4c19b(▶ 0.000; for PageTransition<dynamic>))] flutter: 2020-12-19 16:02:06.560730: SimpleBlocDelegate INFO: onTransition Transition { currentState: BackendDataLoaded, event: UpdateCashGame, nextState: BackendActionInProgressWithData } flutter: 2020-12-19 16:02:06.878949: SimpleBlocDelegate INFO: onTransition Transition { currentState: BackendActionInProgressWithData, event: UpdateCashGame, nextState: UserDataUpdateCompleted } flutter: 2020-12-19 16:02:06.901741: MainApp INFO: AppNavigationObserver didPush from ===||CashGamesNewGameScreen||=== [PageTransition<dynamic>(RouteSettings("CashGamesNewGameScreen", null), animation: AnimationController#4c19b(⏭ 1.000; paused; for PageTransition<dynamic>))] to ===||CashGamesGameDetailsScreen||=== [PageTransition<dynamic>(RouteSettings("CashGamesGameDetailsScreen", null), animation: AnimationController#e1937(▶ 0.000; for PageTransition<dynamic>))] flutter: 2020-12-19 16:02:06.912362: SimpleBlocDelegate INFO: onTransition Transition { currentState: UserDataUpdateCompleted, event: UpdateCashGame, nextState: BackendDataLoaded } VMServiceFlutterDriver: waitFor message is taking a long time to complete... 00:32 +1 ~1 -1: Integration tests cash games flow [E] TimeoutException after 0:00:30.000000: Test timed out after 30 seconds. See https://pub.dev/packages/test#timeouts package:test_api/src/backend/invoker.dart 318:28 Invoker._handleError.<fn> dart:async/zone.dart 1178:47 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run package:test_api/src/backend/invoker.dart 316:10 Invoker._handleError package:test_api/src/backend/invoker.dart 272:9 Invoker.heartbeat.<fn>.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run package:test_api/src/backend/invoker.dart 271:38 Invoker.heartbeat.<fn> dart:async-patch/timer_patch.dart 18:15 Timer._createTimer.<fn> dart:isolate-patch/timer_impl.dart 395:19 _Timer._runTimers dart:isolate-patch/timer_impl.dart 426:5 _Timer._handleMessage dart:isolate-patch/isolate_patch.dart 184:12 _RawReceivePortImpl._handleMessage 00:32 +1 ~2 -1: Integration tests (tearDownAll) flutter: Handling request from Flutter driver: quit_app 00:32 +1 ~2 -1: Integration tests cash games flow [E] DriverError: Failed to fulfill WaitFor due to remote error Original error: ext.flutter.driver: (-32000) Service connection disposed Original stack trace: dart:async/future_impl.dart 23:44 _Completer.completeError package:vm_service/src/vm_service.dart 1972:16 VmService.dispose.<fn> dart:collection-patch/compact_hash.dart 387:8 _LinkedHashMapMixin.forEach package:vm_service/src/vm_service.dart 1970:17 VmService.dispose package:flutter_driver/src/driver/vmservice_driver.dart 528:20 VMServiceFlutterDriver.close test_driver/app_test.dart 32:21 main.<fn>.<fn> ===== asynchronous gap =========================== dart:async/zone.dart 1118:19 _CustomZone.registerUnaryCallback dart:async-patch/async_patch.dart 40:23 _asyncThenWrapperHelper test_driver/app_test.dart main.<fn>.<fn> dart:async/future.dart 226:31 new Future.sync package:test_api/src/util/test.dart 21:12 errorsDontStopTest.<fn> package:test_api/src/backend/invoker.dart 231:15 Invoker.waitForOutstandingCallbacks.<fn> package:test_api/src/backend/invoker.dart 228:14 Invoker.waitForOutstandingCallbacks.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/invoker.dart 228:5 Invoker.waitForOutstandingCallbacks package:test_api/src/util/test.dart 20:20 errorsDontStopTest package:test_api/src/backend/declarer.dart 358:19 Declarer._tearDownAll.<fn>.<fn>.<fn> package:test_api/src/backend/declarer.dart 356:44 Declarer._tearDownAll.<fn>.<fn>.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/invoker.dart 248:12 Invoker.unclosable package:test_api/src/backend/declarer.dart 356:33 Declarer._tearDownAll.<fn>.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/declarer.dart 355:14 Declarer._tearDownAll.<fn> package:test_api/src/backend/invoker.dart 231:15 Invoker.waitForOutstandingCallbacks.<fn> package:test_api/src/backend/invoker.dart 228:14 Invoker.waitForOutstandingCallbacks.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/invoker.dart 228:5 Invoker.waitForOutstandingCallbacks package:test_api/src/backend/invoker.dart 383:17 Invoker._onRun.<fn>.<fn>.<fn> ===== asynchronous gap =========================== dart:async/zone.dart 1118:19 _CustomZone.registerUnaryCallback dart:async-patch/async_patch.dart 40:23 _asyncThenWrapperHelper package:test_api/src/backend/invoker.dart Invoker._onRun.<fn>.<fn>.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/invoker.dart 370:9 Invoker._onRun.<fn>.<fn> package:test_api/src/backend/invoker.dart 415:15 Invoker._guardIfGuarded package:test_api/src/backend/invoker.dart 369:7 Invoker._onRun.<fn> package:stack_trace/src/chain.dart 94:24 Chain.capture.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:stack_trace/src/chain.dart 92:12 Chain.capture package:test_api/src/backend/invoker.dart 368:11 Invoker._onRun package:test_api/src/backend/live_test_controller.dart 153:11 LiveTestController.run dart:async/future.dart 204:37 new Future.microtask.<fn> dart:async/zone.dart 1178:47 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 994:7 _CustomZone.runGuarded dart:async/zone.dart 1034:23 _CustomZone.bindCallbackGuarded.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 994:7 _CustomZone.runGuarded dart:async/zone.dart 1034:23 _CustomZone.bindCallbackGuarded.<fn> dart:async/schedule_microtask.dart 41:21 _microtaskLoop dart:async/schedule_microtask.dart 50:5 _startMicrotaskLoop dart:isolate-patch/isolate_patch.dart 120:13 _runPendingImmediateCallback dart:isolate-patch/timer_impl.dart 402:11 _Timer._runTimers dart:isolate-patch/timer_impl.dart 426:5 _Timer._handleMessage dart:isolate-patch/isolate_patch.dart 184:12 _RawReceivePortImpl._handleMessage package:flutter_driver/src/driver/vmservice_driver.dart 322:7 VMServiceFlutterDriver.sendCommand ===== asynchronous gap =========================== dart:async/zone.dart 1126:19 _CustomZone.registerBinaryCallback dart:async-patch/async_patch.dart 51:8 _asyncErrorWrapperHelper package:test_api/src/backend/invoker.dart Invoker.waitForOutstandingCallbacks.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/invoker.dart 228:5 Invoker.waitForOutstandingCallbacks package:test_api/src/backend/invoker.dart 383:17 Invoker._onRun.<fn>.<fn>.<fn> ===== asynchronous gap =========================== dart:async/zone.dart 1118:19 _CustomZone.registerUnaryCallback dart:async-patch/async_patch.dart 40:23 _asyncThenWrapperHelper package:test_api/src/backend/invoker.dart Invoker._onRun.<fn>.<fn>.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:test_api/src/backend/invoker.dart 370:9 Invoker._onRun.<fn>.<fn> package:test_api/src/backend/invoker.dart 415:15 Invoker._guardIfGuarded package:test_api/src/backend/invoker.dart 369:7 Invoker._onRun.<fn> package:stack_trace/src/chain.dart 94:24 Chain.capture.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 1626:10 _runZoned dart:async/zone.dart 1546:10 runZoned package:stack_trace/src/chain.dart 92:12 Chain.capture package:test_api/src/backend/invoker.dart 368:11 Invoker._onRun package:test_api/src/backend/live_test_controller.dart 153:11 LiveTestController.run dart:async/future.dart 204:37 new Future.microtask.<fn> dart:async/zone.dart 1178:47 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 994:7 _CustomZone.runGuarded dart:async/zone.dart 1034:23 _CustomZone.bindCallbackGuarded.<fn> dart:async/zone.dart 1186:13 _rootRun dart:async/zone.dart 1090:19 _CustomZone.run dart:async/zone.dart 994:7 _CustomZone.runGuarded dart:async/zone.dart 1034:23 _CustomZone.bindCallbackGuarded.<fn> dart:async/schedule_microtask.dart 41:21 _microtaskLoop dart:async/schedule_microtask.dart 50:5 _startMicrotaskLoop dart:isolate-patch/isolate_patch.dart 120:13 _runPendingImmediateCallback dart:isolate-patch/timer_impl.dart 402:11 _Timer._runTimers dart:isolate-patch/timer_impl.dart 426:5 _Timer._handleMessage dart:isolate-patch/isolate_patch.dart 184:12 _RawReceivePortImpl._handleMessage 00:32 +1 ~2 -1: Some tests failed. Unhandled exception: Dummy exception to set exit code. Failed to stop app
Test code:
import 'dart:io'; import 'package:flutter_driver/flutter_driver.dart'; import 'package:intl/intl.dart'; import 'package:test/test.dart'; const String appDateFormatFullMonth = "MMMM dd, yyyy"; final DateFormat appDateFormatterFullMonth = DateFormat(appDateFormatFullMonth); void main() { group('Integration tests', () { FlutterDriver driver; setUpAll(() async { driver = await FlutterDriver.connect(); // Wait for the first frame to be rasterized during the app launch. await driver.waitUntilFirstFrameRasterized(); }); tearDownAll(() async { await driver.requestData("quit_app"); await driver?.close(); }); Future<bool> isPresent(SerializableFinder byValueKey, {Duration timeout = const Duration(seconds: 1)}) async { try { await driver.waitFor(byValueKey, timeout: timeout); return true; } catch (exception) { return false; } } setUp(() async => await driver.requestData("navigate_to_root")); tearDown(() async {}); // TODO - cleanup user data? test('cash games flow', () async { final welcomeButtonUseAnon = find.byValueKey('welcome-useAnon'); expect(await isPresent(welcomeButtonUseAnon), true, reason: "Unable to find welcome useAnon"); await driver.tap(welcomeButtonUseAnon); final mainMenuButtonCashGames = find.byValueKey('mainMenu-cashGames'); expect(await isPresent(mainMenuButtonCashGames), true, reason: "Unable to find Cash Games on main menu button"); await driver.tap(mainMenuButtonCashGames); // cash games welcome screen final cashGamesButtonAddSession = find.byValueKey('cashGames-addSession'); final cashGamesButtonUpdateSession = find.byValueKey('cashGames-updateSession'); expect(await isPresent(cashGamesButtonAddSession), true, reason: "Unable to find Cash Games add session button"); expect(await isPresent(cashGamesButtonUpdateSession), true, reason: "Unable to find Cash Games Update Session button"); // Next screen - new game await driver.tap(cashGamesButtonAddSession); expect(await isPresent(find.text("New game")), true, reason: "Unable to locate new cash game title"); final venueTextField = find.byValueKey('cashGames-new-venue'); final dateTextField = find.byValueKey('cashGames-new-date'); var cancelButton = find.byValueKey('cashGames-new-cancelBtn'); var nextButton = find.byValueKey('cashGames-new-nextBtn'); expect(await isPresent(cancelButton), true); // Fill out new game info var testVenueName = 'Restaurant at End of the world'; expect(await isPresent(venueTextField), true); await driver.tap(venueTextField); await driver.enterText(testVenueName); await driver.waitFor(find.text(testVenueName)); expect(await isPresent(dateTextField), true); await driver.tap(dateTextField); await driver.enterText(appDateFormatterFullMonth.format(DateTime.now())); expect(await isPresent(nextButton), true); await driver.tap(nextButton); // Next screen - Game details await driver.waitFor(find.text(testVenueName)); await driver.waitFor(find.text("Game type:")); await driver.waitFor(find.byValueKey("cashGames-details-cancelBtn")); nextButton = find.byValueKey("cashGames-details-startSessionBtn"); await driver.waitFor(nextButton); await driver.tap(nextButton); // Next screen - ... await driver.waitFor(find.text('Coming soon')); // on the next screen }); }); }
flutter doctor -v
[✓] Flutter (Channel beta, 1.25.0-8.1.pre, on macOS 11.1 20C69 darwin-arm, locale en-US) • Flutter version 1.25.0-8.1.pre at /Users/agautam/tools/flutter • Framework revision 8f89f6505b (4 days ago), 2020-12-15 15:07:52 -0800 • Engine revision 92ae191c17 • Dart version 2.12.0 (build 2.12.0-133.2.beta) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3) • Android SDK at /Users/agautam/Library/Android/sdk • Platform android-30, build-tools 30.0.3 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 12.3) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 12.3, Build version 12C33 • CocoaPods version 1.10.0 [✓] Android Studio • Android Studio at /Applications/Android Studio 4.2 Preview.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6842174) [✓] Android Studio (version 4.1) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495) [✓] VS Code (version 1.52.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.17.0 [✓] Connected device (1 available) • iPhone 12 Pro Max (mobile) • D52F7EE5-34D4-4444-B05E-743F72B240A9 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator) • No issues found!
Thanks
Ajay
-
M. Leonhard over 2 yearsRelated: "Integration tests to produce useful failure messages" github.com/flutter/flutter/issues/88591
-
-
Ajay Gautam over 3 yearsYeah... thats what I ended up doing. Its ugly though :(