xUnit - Display test names for theory memberdata (TestCase)

Question

I've been using NUnit for testing and I'm really fond of test cases. In NUnit you can easily set each test name in the test case using the SetName function in a TestCaseData class.

Does xUnit have a similar function for this?

Currently I can only see one test in the test explorer even tho I have 6 tests in the test case.

xUnit test

public class LogHandler : TestBase
{
    private ILogger _logger;

    public LogHandler()
    {
        //Arrange
        LogAppSettings logAppSettings = GetAppSettings<LogAppSettings>("Log");

        IOptions<LogAppSettings> options = Options.Create(logAppSettings);

        LogService logService = new LogService(new Mock<IIdentityService>().Object, options);

        LogProvider logProvider = new LogProvider(logService);

        _logger = logProvider.CreateLogger(null);
    }

    public static IEnumerable<object[]> TestCases => new[]
    {
        new object[] { LogLevel.Critical,
            new EventId(),
            new Exception(),
            1 },
        new object[] { LogLevel.Error,
            new EventId(),
            new Exception(),
            1 },
        new object[] { LogLevel.Warning,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.Information,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.Debug,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.Trace,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.None,
            new EventId(),
            new Exception(),
            0 }
    };

    [Theory, MemberData(nameof(TestCases))]
    public void Test(LogLevel logLevel, EventId eventId, Exception exception, int count)
    {
        //Act
        _logger.Log<object>(logLevel, eventId, null, exception, null);

        //Assert
        int exceptionCount = Database.Exception.Count();

        Assert.Equal(exceptionCount, count);
    }
}

xUnit test window

Should be 6 tests here instead of one! (ignore GetOrganisationStatuses).

NUnit test case

public static IEnumerable TestDatabaseCases
{
    get
    {
        yield return new TestCaseData(LogLevel.Critical,
            new EventId(1),
            new Exception("Exception"),
            0,
            1).SetName("InsertException_Should_Insert_When_LogLevel_Critical");

        yield return new TestCaseData(LogLevel.Error,
            new EventId(1),
            new Exception("Exception"),
            0,
            1).SetName("InsertException_Should_Insert_When_LogLevel_Error");

        yield return new TestCaseData(LogLevel.Warning,
            new EventId(1),
            new Exception("Exception"),
            0,
            0).SetName("InsertException_Should_Not_Insert_When_LogLevel_Warning");

        yield return new TestCaseData(LogLevel.Information,
            new EventId(1),
            new Exception("Exception"),
            0,
            0).SetName("InsertException_Should_Not_Insert_When_LogLevel_Information");

        yield return new TestCaseData(LogLevel.Debug,
            new EventId(1),
            new Exception("Exception"),
            0,
            0).SetName("InsertException_Should_Not_Insert_When_LogLevel_Debug");
    }
}

NUnit test window

This is what I want in xUnit!

How do I in xUnit set a name for each test in the test case?

Solution 1

Currently I can only see one test in the test explorer even tho I have 6 tests in the test case.

This is because your test data is not considered serializable by xUnit.net. See this issue: https://github.com/xunit/xunit/issues/1473

The salient details are:

The short answer

If some of your theory data can't be "serialized" by xUnit.net, then it cannot be encapsulated into the serialization of a test case which we're required to do for the Visual Studio test runner.

The long answer

In the Visual Studio test runner, test cases are discovered in one process, and executed in another. Therefore, test cases must be able to be turned into an unqualified string representation (aka, "serialized") in order to be run. We can also serialize at the test method level, because that just involves knowing the type and method name (both strings). When you start putting data into the mix, we need to ensure we know how to serialize that data; if we can't serialize all of the data for a theory, then we have to fall back to just a single method (which we know we can serialize).

The full answer contains more detail.

In NUnit you can easily set each test name in the test case using the SetName function in a TestCaseData class.

Does xUnit have a similar function for this?

Not at this time.

Solution 2

This worked for me:

public class TestScenario
{
...
    public override string ToString()
    {
        return $"Role: {Role}...";
    }
}

    [Theory]
    [ClassData(typeof(MyProvider))]
    public void TestScenarios(TestScenario scenaro)
    {
        TestScenarioInternal(scenaro);

    }

Solution 3

Actually there is a working solution requiring some plumbing code which should work with unchanged tests. It requires implementation of custom TheoryAttribute, custom TheoryDiscoverer, and custom TestCase classes. Whole solution is available under MIT license in this repo DjvuNet/DjvuNet.Shared.Tests.

Required files with implementations are: DjvuTheoryAttribute, DjvuTheoryDiscoverer, DjvuNamedDataRowTestCase, DjvuDataRowTestCase

Usage is straightforward: compile above files either including them directly into test assembly or as a separate assembly and use them in code as follows:

    [DjvuTheory]
    [ClassData(typeof(DjvuJsonDataSource))]
    public void DirmChunk_Theory(DjvuJsonDocument doc, int index)
    {
        int pageCount = 0;
        using (DjvuDocument document = DjvuNet.Tests.Util.GetTestDocument(index, out pageCount))
        {
            DjvuNet.Tests.Util.VerifyDjvuDocument(pageCount, document);
            DjvuNet.Tests.Util.VerifyDjvuDocumentCtor(pageCount, document);

            // DirmChunk is present only in multi page documents
            // in which root form is of DjvmChunk type
            if (document.RootForm.ChunkType == ChunkType.Djvm)
            {
                DirmChunk dirm = ((DjvmChunk)document.RootForm).Dirm;

                Assert.NotNull(dirm);

                Assert.True(dirm.IsBundled ? doc.Data.Dirm.DocumentType == "bundled" : doc.Data.Dirm.DocumentType == "indirect");

                var components = dirm.Components;
                Assert.Equal<int>(components.Count, doc.Data.Dirm.FileCount);
            }
        }
    }

One of the theory function arguments is not serializable in xUnit but despite that theory tests will be displayed individually and numbered. If first argument to theory function is of the string type it will be used as a name of the test besides being an argument to function invocation.

Credit for the idea goes to other developer - I have to find a link to his code - but it was reimplemented from scratch for DjvuNet project.

Comments

[Reft]

That's a shame.. As it stands now I'll continue with NUnit. Hopefully they will fix this.

[Jacek Blaszczynski]

@Reft Common it's not a shame but small limitation which can be easily compensated for by small piece of code - see below for my answer. Anyway xUnit is a test framework of choice for large number of most important in DotNet world repos.

[lbrahim]

@JacekBlaszczynski Can you comment on what those reasons might be?