How to run tests in a playlist file from mstest command line

Question

Is there a way I can use the playlist file to run tests from commandline using MSTest.exe? I tried using following command line but it fails with and error -

mstest.exe /testmetadata:test.playlist

The file 'test.playlist' has unknown format and cannot be converted to the current version.

Note that mstest version is 11.0.50727.1 and I am using VS 2012

my playlist file just contains couple of XML elements

<Playlist Version="1.0">
  <Add Test="MyTest" />
  <Add Test="AnotherTest" />
</Playlist>

Solution 1

mstest.exe has been deprecated. For Visual Studio 2012 SP1 and above we use vstest.console.exe. It still won't run your playlist file. There is a feature request open with Microsoft.

We're in a bad state right now with test lists also deprecated. The only way I can see to discriminate between different tests is to use Test Categories

Another option would be to switch to a different unit testing framework.

I wish there were a better answer.

Solution 2

As Nate Zaugg mentioned, there isn't currently a way to do it, but it's fairly easy to generate a list of tests programmatically.

The playlist file is just XML. It has a root node with child nodes. You can read the "Test" attribute of each node to get the FullyQualifiedNames of each of the tests you want to run. You can probably strip out the namespace and class before the test names at this point.

If you're able to use vstest.console.exe (which you should, with VS 2012), you can follow the instructions below. vstest.console.exe has the optional parameter "TestCaseFilter" which you can use to call out each test mentioned in the playlist. It's not particularly elegant, but it should work.

So, for some generic tests "MethodName1", "MethodName2", and "MethodName3", that are in the myTestFile.dll and myOtherTestFile.dll you would generate the following command:

vstest.console.exe myTestFile.dll myOtherTestFile.dll /Tests:MethodName1,MethodName2,MethodName3"

Solution 3

You could use .orderedtest instead of .playlist

Ordered tests can be created and edited in VS2013. The format is otherwise similiar to .playlist but it contains links to test GUIDs so its more complicated to modify programmatically.

From command line, run with:

MSTest.exe /testcontainer:mylist.orderedtest

Not sure if it works in VS2012.

Solution 4

It looks like with VSTest we now have a TestCaseFilter option with TestCategory and Priority. I'm still looking into it, but hopefully this helps someone else.

View more solutions

Comments

[Ben Power]

I'd like to know how you got on. I'm seriously considering moving to another framework too, MS testing is clunky as heck even when it works perfectly, which it often doesn't.

[Nate Zaugg]

We used Test Categories and just ran with the DLL's and the test categories as the parameters. It's stupid! There are other problems with MS Test when using it with a DI framework like unity. Unless there is a hard reference, MS Test thinks it knows better and doesn't copy that reference over to the target test directory so you're left without all of the references you need.

[Kevin Holt]

We take a similar approach, and just programmatically generate a batch file to launch VsTest with a huge "/Tests:xxx" argument that we build on the fly. Ugly, but works fine.

[TommyD]

@Kevin This is not only ugly, this can also be problematic considering that the maximum command's length is 12190 bytes for a Powershell script block and 8191 characters for cmd.exe. This is another bad design decision from Microsoft that makes our life harder for no good reasons.