Xcode 4: Run tests from the command line (xcodebuild)?

Question

I've created a brand new iOS project in Xcode 4, and included unit tests. The default app has 2 targets, the main application and the unit test bundle. Using "Product > Test" (Command-U) builds the application, builds the unit test bundle, launches the iOS simulator and runs the tests. Now I'd like to be able to do the same thing from the command line. The command line tool (xcodebuild) doesn't have a "test" action, but it seems like I should be able to build the unit test bundle target directly, since it depends on the application itself. However, running:

xcodebuild -target TestAppTests -sdk iphonesimulator4.3 -configuration Debug build

gives the following message:

/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:95: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

That seems like a lie, since Test Host is set for my unit test bundle target when I run Command-U from the GUI. I've seen previous posts about the separation between logic tests and application tests, but it seems like Xcode 4 does away with that distinction. Any clue how I can run my tests from the command line?

Solution 1

Important Note

With Xcode 5.1 (perhaps earlier Xcode as well) test is a valid build action.

We were able to replace the entire hack below with a call to xcodebuild using the build action of test and with appropriate -destination options. man xcodebuild for more info.

The information below is left here for posterity

---

I tried hacking Apple's scripts to run unit tests as mentioned in

Running Xcode 4 unit tests from the command line

and

Xcode4: Running Application Tests From the Command Line in iOS

and numerous similar postings across the web.

However, I ran into a problem with those solutions. Some of our unit tests exercised the iOS Keychain and those calls, when running in the environment that comes from hacking Apple's scripts, failed with an error (errSecNotAvailable[-25291] for the morbidly curious). As a result, the tests always failed... an undesirable feature in a test.

I tried a number of solutions based on information I found elsewhere on the web. Some of those solutions involved trying to launch the iOS simulator's security services daemon, for example. After struggling with those, My best bet seemed to be to run in the iOS simulator with the full benefit of the simulator's environment.

What I did, then was get ahold of the iOS Simulator launching tool ios-sim. This command line tool uses private Apple frameworks to launch an iOS application from the command line. Of particular use to me, however, was the fact that it allows me to pass both Environment Variables and Command Line Arguments to the app that it is launching.

Though the Environment variables, I was able to get my Unit Testing bundle injected into my Application. Through the command line arguments, I can pass the "-SenTest All" needed to get the app to run the unit tests and quit.

I created a Scheme (which I called "CommandLineUnitTests") for my unit testing bundle and checked the "Run" action in the build section as described in the posts above.

Rather than hacking Apple's scripts, though, I replaced the script with one that launches the application using ios-sim and sets up the environment to inject my unit testing bundle into the application separately.

My script is written in Ruby which is more familiar to me than BASH scripting. Here's that script:

if ENV['SL_RUN_UNIT_TESTS'] then
    launcher_path = File.join(ENV['SRCROOT'], "Scripts", "ios-sim")
    test_bundle_path= File.join(ENV['BUILT_PRODUCTS_DIR'], "#{ENV['PRODUCT_NAME']}.#{ENV['WRAPPER_EXTENSION']}")

    environment = {
        'DYLD_INSERT_LIBRARIES' => "/../../Library/PrivateFrameworks/IDEBundleInjection.framework/IDEBundleInjection",
        'XCInjectBundle' => test_bundle_path,
        'XCInjectBundleInto' => ENV["TEST_HOST"]
    }

    environment_args = environment.collect { |key, value| "--setenv #{key}=\"#{value}\""}.join(" ")

    app_test_host = File.dirname(ENV["TEST_HOST"])
    system("#{launcher_path} launch \"#{app_test_host}\" #{environment_args} --args -SenTest All #{test_bundle_path}")
else
    puts "SL_RUN_UNIT_TESTS not set - Did not run unit tests!"
end

Running this from the command line looks like:

xcodebuild -sdk iphonesimulator -workspace iPhoneApp.xcworkspace/ -scheme "CommandLineUnitTests" clean build SL_RUN_UNIT_TESTS=YES

After looking for the SL_RUN_UNIT_TESTS environment variable, the script finds the "launcher" (the iOS-sim executable) within the project's source tree. It then constructs the path to my Unit Testing Bundle based on build settings that Xcode passes in environment variables.

Next, I create the set of runtime Environment Variables for my running application that inject the unit testing bundle. I set up those variables in the environment hash in the middle of the script then use some ruby grunge to join them into a series of command line arguments for the ios-sim application.

Near the bottom I grab the TEST_HOST from the environment as the app I want to launch and the system command actually executes ios-sim passing the application, the command arguments to set up the environment, and the arguments -SenTest All and the test bundle path to the running application.

The advantage of this scheme is that it runs the unit tests in the simulator environment much as I believe Xcode itself does. The disadvantage of the scheme is that it relies on an external tool to launch the application. That external tool uses private Apple frameworks, so it may be fragile with subsequent OS releases, but it works for the moment.

P.S. I used "I" a lot in this post for narrative reasons, but a lot of the credit goes to my partner in crime, Pawel, who worked through these problems with me.

Solution 2

I was inspired by Jonah's post and found a way to do it:

http://longweekendmobile.com/2011/04/17/xcode4-running-application-tests-from-the-command-line-in-ios/

Basically, you need Xcode 4 and you have to hack a script to make it work, but it does.

The key point is convincing Xcode 4 to run your iOS test bundle as if it was a MacOS X bundle - it's a problem with the platform, Xcode doesn't want to run application tests out of the box on the command line. Funny, because it seems to work.

There's also a sample project on the site.

Solution 3

what you're looking for is this undocumented argument (you do need sdk and target too) to run your OCUnit Tests from the terminal

xcodebuild  -target MyTarget -sdk iphonesimulator   TEST_AFTER_BUILD=YES

Solution 4

It's an incomplete solution but I was able to run command line builds of logic tests in their own scheme and build target: http://blog.carbonfive.com/2011/04/06/running-xcode-4-unit-tests-from-the-command-line/

Solution 5

xctool solves this issue: https://github.com/facebook/xctool

we use it on our continuous integration server without problems

View more solutions

Comments

[Bob Whiteman]

The instructions on that page worked for me, and I didn't have to change anything in /Developer. Thanks!

[Stew]

I just tried @makdad's solution and it worked for me. Surely there is an easier way to run command line tests by now without hacking /Developer scripts?

[makdad]

If you can find it, I'll link to your blog! :)

[makdad]

Bob, what version of iOS are you on? Or more specifically - Xcode?

[otto]

Any news on this issue? I'm still interested in running application tests from the command line without hacking into /Developer. My first try with Xcode 4.2 seems to encounter the same problem.

[makdad]

@otto, no not yet :) people on my blog's comments have been saying that as of Xcode 4.2, the hack is still necessary.

[makdad]

@BobWhiteman, also if you were just running Logic tests, then this hack wouldn't be necessary - was probably your case, no?

[Scott Thompson]

Incidentally the DYLD_INSERT_LIBRARIES is the same relative path in my script as it is in the application environment when you launch your application's Test action from the IDE. I believe this path is relative to another environment variable "DYLD_ROOT". Just didn't want it to seem like there was deep magic there.

[Admin]

I didn't hack the script in PLATFORM_DEVELOPER_TOOLS_DIR. Instead I copied it into my repo, tweaked it there and created a wrapper script. But yeah it's annoying that this is needed.

[makdad]

@GrahamLee I've started doing the same. I think that's the best practice here now...

[makdad]

Scott, yes, I had similar problems with the keychain, but I didn't need to go to these depths to fix it. Did you see this post: stackoverflow.com/questions/9996578/…

[Sulthan]

Finally somebody who found the correct solution, without hacking scripts - just duplicating XCode's implementation.

[makdad]

@ScottThompson Well, finally I found the need to go to these lengths (so thank you for doing this research). Apparently what user you are running as makes a difference w/r/t to the keychain (obviously); we needed to change that on our Jenkins setup, so this setup you've offered here is probably what I need to adopt. Since my blog article on the topic is pretty popular I'll have to update it and give your team some credit if I can get it to work properly.

[ThomasW]

@bigkm's answer worked for me and it is much simpler. It may depend on the version of Xcode.

[ThomasW]

@bigkm's answer worked for me and it is much simpler. It may depend on the version of Xcode.

[ThomasW]

This works for me with Xcode 4.4.1 and 4.5. It didn't work on 4.3, but that might be due to an unrelated issue.

[lawicko]

This answer should be accepted, works in XCode 4.5 and with Jenkins too! Thanks for sharing. bigkm's answer will only run logic unit tests and cannot be applied if you build a workspace and schemes instead of targets.

[lawicko]

This only runs the logic unit tests and may be problematic if you build workspace instead of targets. In my project I have the workspace take care of the dependencies so it's not really possible for me to switch to building a target.

[makdad]

Side note that this is not an undocumented argument, but rather an environment variable you are setting.

[Sulthan]

It's a shame I cannot share my whole jenkins solution (company property), anyway - WaxSim project on github (see the latest clone) provides better control than ios-sim over the simulator framework - you can pass the parameters more easily. I can also recommend to see how gcov command line utility works. It is very easy to generate pretty code coverage reports. You can also use sed to skip non-testable code delimited by COV_NF_START/_END/_LINE if you use CoverStory like me (the trick is to replace ##### by something else).

[makdad]

Just an update - this works in Xcode 4.5 and I am going to update the blog post referenced above to give some credit to this solution, as a lot of people come to my blog looking for solutions to this...

[Miroslav Kovac]

I always get the following error when running test from command line: [DEBUG] Session could not be started: Error Domain=DTiPhoneSimulatorErrorDomain Code=1 "Unknown error." UserInfo=0x7ffc23ce0dd0 {NSLocalizedDescription=Unknown error., DTiPhoneSimulatorUnderlyingErrorCodeKey=-1}. Has anyone experiencing same problem?

[Ciryon]

Miroslav, yeah I did get the same problem, but it helped to shut down the iOS Simulator which seems to have been started from Xcode previously.

[Ciryon]

Scott, the only problem I've seen is that it seems the error code from running the script isn't correct. If the tests fail it still says "build succeeded". I found a solution by adding exit($?.exitstatus) to the row below system(...)

[Miroslav Kovac]

@Ciron, I'm afraid that the error is not from already started iOS Simulator. I restarted computer and run the command line without even starting Xcode or iOS Simulator, but the error is still there. I'll try to setup ENV variables and call the ios-sim directly from command line.

[kraftydevil]

Where should I expect to see testing output? Everything builds great but there is no output from my tests. I'm doing an STFail just to make sure something fails... nothing seems to happen during the 'launch'

[Maciej Trybiło]

@ScottThompson How did you exactly use the Ruby script? It has to replace /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneS‌​imulator.platform/De‌​veloper/Tools/RunPla‌​tformUnitTests? It has to be called by it? I'm a bit lost about where it comes in.

[Joel S]

@MiroslavKovac - bumped into the same issue and this answer seemed to do the trick stackoverflow.com/a/13006893/83592

[Raphael Oliveira]

I configured exactly like the site but Im unable to run the tests. The LogicTest scheme build successfully but any test is being run.

[aledalgrande]

I get warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set). with XCode 4.6.1 when I run xcodebuild -target TestTarget -sdk iphonesimulator TEST_AFTER_BUILD=YES

[Segfault]

I replaced the script with one that launches the application using ios-sim Ack, need more details on this part. How is this done?

[meaning-matters]

I'm trying xctool but still get the very error message this post is about. Which version of Xcode are you using?

[AnneTheAgile]

@meaning-matters, their release notes show 2013 was its first release, and changes indicate that xcode5 was the first supported version. It's still actively developed and looks like a nice api. Ty skrusche for the tip, I came here for xcode 6 and this is very helpful. github.com/facebook/xctool/releases