Error with XCTestExpectation: API violation - multiple calls made to -[XCTestExpectation fulfill]

Question

I'm using XCTestExpectations in Xcode 6 (Beta 5) for asynchronous testing. All my asynchronous tests pass individually every time I run them. However, when I try to run my entire suite, some tests do not pass, and the app crashes.

The error I get is says API violation - multiple calls made to -[XCTestExpectation fulfill]. Indeed, this is not true within a single method; my general format for my tests is shown below:

- (void) someTest {
    /* Declare Expectation */
    XCTestExpectation *expectation = [self expectationWithDescription:@"My Expectation"];
    [MyClass loginOnServerWithEmail:@"[email protected]" andPassword:@"asdfasdf" onSuccess:^void(User *user) {
        /* Make some assertions here about the object that was given. */

        /* Fulfill the expectation */
        [expectation fulfill];
    }];

    [self waitForExpectationsWithTimeout:5.0 handler:^(NSError *error) {
        /* Error handling here */
    }];
}

Again, these tests do pass when run individually, and they are actually making network requests (working exactly as intended), but together, the collection of tests fail to run.

I was able to have a look at this post here, but was unable to get the solution to work for me.

Additionally, I'm running OSX Mavericks and using Xcode 6 (Beta 5).

Solution 1

I don't think using __weak or __block is a good approach. I have written many unit tests using XCTestExpectation for awhile and never had this problem until now. I finally found out that real cause of the problem which potentially may cause bugs in my app. The root cause of my problem is that startAsynchronousTaskWithDuration calls the completionHandler multiple time. After I fix it the API violation went away!

[self startAsynchronousTaskWithDuration:4 completionHandler:^(id result, NSError *error) {
    XCTAssertNotNil(result);
    XCTAssertNil(error);
    [expectation fulfill];
}];

Although it took me a few hours to fix my unit tests but I came to appreciate the API violation error which will help me avoid future runtime problem in my app.

Solution 2

Error: API violation - multiple calls made to -[XCTestExpectation fulfill]

I got the same error when set expectation.expectedFulfillmentCount.

expectation.assertForOverFulfill = false

If expectation.assertForOverFulfill = true and fulfill() is called when it is already fulfilled then it throws an exception

Solution 3

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - multiple calls made to -[XCTestExpectation fulfill] for whatever.'

I get the error above with the following code:

func testMultipleWaits() {
    let exp = expectation(description: "whatever")
    for _ in 0...10 {
        DispatchQueue.main.async {
            exp.fulfill()
        }
        wait(for: [exp], timeout: 1)
    }
}

Basically a given expectation can be fulfilled once, it can also be waited once.

Meaning the following change would still not fix it, because then you've still waited multiple times for it. It would give you the following error.

failed: caught "NSInternalInconsistencyException", "API violation - expectations can only be waited on once, whatever have already been waited on"

func testMultipleWaits() {
    let exp = expectation(description: "whatever")
    for i in 0...10 {
        DispatchQueue.main.async {
            if i == 6 {
                exp.fulfill()
            }
        }
        wait(for: [exp], timeout: 1)
    }
}

What makes it confusing is that while the above change still crashes, the test does get finished and gives you a failure of "Asynchronous wait failed: Exceeded timeout of 1 seconds, with unfulfilled expectations: "whatever".'

It's misleading because you may spend time fixing the test, all while you should be spending time fixing the crash which is the root cause

---

The fix here is set the expectation inside the for-loop. This way the expectation is fulfilled and waited against once per expectation.

func testMultipleWaits() {        
    for _ in 0...10 {
        let exp = expectation(description: "whatever")
        DispatchQueue.main.async {
            exp.fulfill()
        }
        wait(for: [exp], timeout: 1)
    }
}

Solution 4

Even though this question is quite old, I don't think it has a proper explanation of the root problems. This is surely hitting one of the problems you can get with XCTestExpectation. For a full deep explanation, go to Jeremy W. Sherman post where he explained XCTestExpectation gotchas perfectly.

Summary:

• Always assign your expectations to a weak reference, and then bail in your callback if it’s nil.
• In the rare case where you expect your callback to be triggered more than once, you can avoid fulfilling by annihilating your weak reference after fulfilling it and then ignoring future calls. More likely, you know how many times you should be called, and you’ll want to fulfill the promise only on the last call. But the workaround is there if you need it.
• If you’re already working with a promise-based API, you can skip XCTestExpectation and use whatever wait-and-see API is provided by that promise instead of XCTest’s own. This has the added advantage of linearizing your test code by eliminating the need to handle the delivered value in the closure (or manually shuttle it out to assert against after the XCTest wait has finished).

Code examples properly avoiding errors and crashes:

func testPreparedForNotWaitingLongEnough() {
    weak var promiseToCallBack = expectationWithDescription("calls back")
    after(seconds: callBackDelay) { () -> Void in
        guard let promise = promiseToCallBack else {
            print("too late, buckaroo")
            return
        }

        print("I knew you'd call!")
        promise.fulfill()
    }

    waitForExpectationsWithTimeout(callBackDelay / 2) { error in
        print("Aww, we timed out: \(error)")
    }
}

func testSafelyDoubleTheFulfillment() {
    weak var promiseToCallBack = expectationWithDescription("calls back")
    let callBackDelay: NSTimeInterval = 1

    twiceAfter(seconds: callBackDelay) {
        guard let promise = promiseToCallBack else {
            print("once was enough, thanks!")
            return
        }

        promise.fulfill()
        promiseToCallBack = nil
    }

    let afterCallBack = 2 * callBackDelay
    waitForExpectationsWithTimeout(afterCallBack, handler: nil)
}

View more solutions

Comments

[James Webster]

Although this is potentially quite a good answer, it is essentially a "link only" answer. You should include some information from your links so that if the information behind the links is ever changed/lost, the answer still makes sense. See this link

[jlngdt]

dude ... your a even not on the same language ... you talk about optional wrapper for a objective-c question ...