Why do some Flutter widget tests fail if executed together but pass individually?

Question

When having multiple tests in the same file, and running one test after the other. Some kinds of tests are failing, but the same tests pass when run individually.

This is my test file at the moment, it's kinda long though:

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:rec/Components/Inputs/text_fields/DniTextField.dart';
import 'package:rec/Helpers/Validators.dart';

import '../../test_utils.dart';

void main() {
  testWidgets('DniTextField works with invalid DNI', (WidgetTester tester) async {
    var key = GlobalKey<DniTextFieldState>();
    var formKey = GlobalKey<FormState>();
    var onChangedResult;

    var widget = Form(
      key: formKey,
      child: DniTextField(
        key: key,
        onChange: (String value) {
          onChangedResult = value;
        },
        validator: (s) => Validators.verifyIdentityDocument(s),
      ),
    );
    await tester.pumpWidget(
      TestUtils.wrapPublicRoute(widget),
    );
    await tester.pumpAndSettle();

    // Test that widget has at least rendered
    TestUtils.widgetExists(widget);

    // Enter text into field with
    var widgetFinder = find.byWidget(widget);
    await tester.tap(widgetFinder);
    await tester.showKeyboard(widgetFinder);
    await tester.enterText(widgetFinder, 'invaliddni');

    await tester.pumpAndSettle();

    expect(onChangedResult, 'invaliddni');

    expect(formKey.currentState.validate(), false);
  });

  testWidgets('DniTextField works with valid DNI', (WidgetTester tester) async {
    var key = GlobalKey<DniTextFieldState>();
    var formKey = GlobalKey<FormState>();
    var onChangedResult;

    var widget = Form(
      key: formKey,
      child: DniTextField(
        key: key,
        onChange: (String value) {
          onChangedResult = value;
        },
        validator: (s) => Validators.verifyIdentityDocument(s),
      ),
    );
    await tester.pumpWidget(
      TestUtils.wrapPublicRoute(widget),
    );
    await tester.pumpAndSettle();

    // Enter text into field
    var widgetFinder = find.byType(DniTextField);
    await tester.tap(widgetFinder);
    await tester.showKeyboard(widgetFinder);
    await tester.enterText(widgetFinder, '80008000k');

    await tester.pumpAndSettle();

    expect(onChangedResult, '80008000k');

    expect(formKey.currentState.validate(), true);
  });

  testWidgets('DniTextField works with valid DNI with trailing space', (WidgetTester tester) async {
    var key = GlobalKey<DniTextFieldState>();
    var formKey = GlobalKey<FormState>();
    var onChangedResult;

    var widget = Form(
      key: formKey,
      child: DniTextField(
        key: key,
        onChange: (String value) {
          onChangedResult = value;
        },
        validator: (s) => Validators.verifyIdentityDocument(s),
      ),
    );
    await tester.pumpWidget(TestUtils.wrapPublicRoute(widget));
    await tester.pumpAndSettle();

    // Enter text into field
    var widgetFinder = find.byWidget(widget);
    await tester.tap(widgetFinder);
    await tester.showKeyboard(widgetFinder);
    await tester.enterText(widgetFinder, '80008000k ');

    await tester.pumpAndSettle();

    // The value emitted by the field, should be free of trailing whitespace
    expect(onChangedResult, '80008000k');

    expect(formKey.currentState.validate(), true);
  });
}

If I run this like this:

$ flutter test test/Components/inputs/dni_text_field_test.dart

The first test passes, but the next 2 don't, spitting out the following error:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test:
The finder "zero widgets with the given widget
(DniTextField-[LabeledGlobalKey<DniTextFieldState>#0fb86]) (ignoring offstage widgets)" (used in a
call to "tap()") could not find any matching widgets.

When the exception was thrown, this was the stack:
#0      WidgetController._getElementPoint (package:flutter_test/src/controller.dart:902:7)
#1      WidgetController.getCenter (package:flutter_test/src/controller.dart:841:12)
#2      WidgetController.tap (package:flutter_test/src/controller.dart:273:18)
#3      main.<anonymous closure> (file:///[masked]/test/Components/inputs/dni_text_field_test.dart:99:18)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

If I then comment out the first test, then the second one passes but the third one does not. As I said the test pass if I run them individually.

I can't find any information about this, not even one thing. Maybe there is someone here that can guide me in the right direction. I'm a bit of a noob with Flutter widget testing, so I might be missing some important things.

The only solution I've found for now is having each test in a separate file. It's not ideal though, would be much better to contain related tests in the same file. Flutter has examples showing that multiple tests per file is allowed as it should be.

Flutter Version: 2.4.0-5.0.pre.87

Solution 1

Okay I found the solution and reason, fixed thanks to this issue and this SO question.

The main issue was the use of Localizations that loaded from JSON files. The solution was to wrap each test with tester.runAsync()

testWidgets('widget test 2', (WidgetTester tester) async {
  await tester.runAsync(() async {
    // tests
  });
});

Solution 2

I suppose this trouble occure when different test are not craeting different instances of widgets (or they are conflicting in some other way), but there is a turnaround that might help you.

If you want to create a single button running all your tests - you can create a bash (or cmd on windows) file, that is running all your tests sequentially.

Example all_tests.cmd

dart test_1.dart
dart test_2.dart
dart test_3.dart

It is not the clearest solution, but it might work for first time.

Comments

[jamesdlin]

If your tests pass individually but not together, that's indicative of your tests not being hermetic: that is, you have some form of state somewhere that is preserved across tests. A minimal, reproducible example would help.

[Keff]

Ohh thanks, makes sense. Though the only thing state-related thing I have is flutter_localizations, could that be causing the issue? I will try set up an example

[Keff]

Found the issue, it had to do with Localizations. Is there any way to reset the state before each of the tests? or how would you recommend approaching this? thanks again!

[Keff]

Me again lol '^^I found a fix in this answer stackoverflow.com/a/52474073/5900163

[Keff]

Yup, that's somewhat been my solution in the past, should've added it to the question maybe. I'm hoping lets me know exactly the issue as I would like to have all related tests in one file. I've to see flutter do it in their examples, so it can be done... Thanks anyways!

[Smundo]

Thank you, that did the trick for me. I have been wasting hours on this issue.