How do I enforce exception message with ExpectedException attribute

Question

I thought these two tests should behave identically, in fact I have written the test in my project using MS Test only to find out now that it does not respect the expected message in the same way that NUnit does.

NUnit (fails):

[Test, ExpectedException(typeof(System.FormatException), ExpectedMessage = "blah")]
public void Validate()
{
    int.Parse("dfd");
}

MS Test (passes):

[TestMethod, ExpectedException(typeof(System.FormatException), "blah")]
public void Validate()
{
    int.Parse("dfd");
}

No matter what message I give the ms test, it will pass.

Is there any way to get the ms test to fail if the message is not right? Can I even create my own exception attribute? I would rather not have to write a try catch block for every test where this occurs.

Solution 1

That mstest second parameter is a message that is printed out when the test fails. The mstest will succeed if a formatexception is thrown. I found this post that may be useful

http://blogs.msdn.com/b/csell/archive/2006/01/13/expectedexception-might-not-be-what-you-ve-expected.aspx

Solution 2

We use this attribute all over the place and we clearly misunderstood the second parameter (shame on us).

However, we definitely have used it to check the exception message. The following was what we used with hints from this page. It doesn't handle globalization, or inherited exception types, but it does what we need. Again, the goal was to simply RR 'ExpectedException' and swap it out with this class. (Bummer ExpectedException is sealed.)

public class ExpectedExceptionWithMessageAttribute : ExpectedExceptionBaseAttribute
{
    public Type ExceptionType { get; set; }

    public string ExpectedMessage { get; set; }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType)
    {
        this.ExceptionType = exceptionType;
    }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType, string expectedMessage)
    {
        this.ExceptionType = exceptionType;
        this.ExpectedMessage = expectedMessage;
    }

    protected override void Verify(Exception e)
    {
        if (e.GetType() != this.ExceptionType)
        {
            Assert.Fail($"ExpectedExceptionWithMessageAttribute failed. Expected exception type: {this.ExceptionType.FullName}. " +
                $"Actual exception type: {e.GetType().FullName}. Exception message: {e.Message}");
        }

        var actualMessage = e.Message.Trim();
        if (this.ExpectedMessage != null)
        {
            Assert.AreEqual(this.ExpectedMessage, actualMessage);
        }

        Debug.WriteLine($"ExpectedExceptionWithMessageAttribute:{actualMessage}");
    }
}

Solution 3

@rcravens is correct - the second param is a message that is printed if the test fails. What I've done to work around this is craft my tests a little differently. Admittedly, I don't love this approach, but it works.

[TestMethod]
public void Validate()
{
    try
    {
        int.Parse("dfd");
    
        // Test fails if it makes it this far
        Assert.Fail("Expected exception was not thrown.");
    }
    catch (ArgumentNullException ex) // <-- Expected Exception class
    {
        Assert.AreEqual("blah", ex.Message);
    }
    catch (Exception ex) // All the other exceptions
    {
        Assert.Fail("Thrown exception was of the wrong type");
    }
}

Solution 4

I extended answer from BlackjacketMack for our project by adding support for contains, case-insensitive and ResourcesType-ResourceName combinations.

Usage example:

public class Foo
{
    public void Bar(string mode)
    {
        if (string.IsNullOrEmpty(mode)) throw new InvalidOperationException(Resources.InvalidModeSpecified);
    }

    public void Bar(int port)
    {
        if (port < 0 || port > Int16.MaxValue) throw new ArgumentOutOfRangeException("port");
    }
}

[TestClass]
class ExampleClass
{
    [TestMethod]
    [ExpectedExceptionWithMessage(typeof(InvalidOperationException), typeof(Samples.Resources), "InvalidModeSpecified")]
    public void Raise_Exception_With_Message_Resource()
    {
        new Foo().Bar(null);
    }

    [TestMethod]
    [ExpectedExceptionWithMessage(typeof(ArgumentOutOfRangeException), "port", true)]
    public void Raise_Exeception_Containing_String()
    {
        new Foo().Bar(-123);
    }
}

Here is the updated class:

public class ExpectedExceptionWithMessageAttribute : ExpectedExceptionBaseAttribute
{
    public Type ExceptionType { get; set; }

    public Type ResourcesType { get; set; }

    public string ResourceName { get; set; }

    public string ExpectedMessage { get; set; }

    public bool Containing { get; set; }

    public bool IgnoreCase { get; set; }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType)
        : this(exceptionType, null)
    {
    }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType, string expectedMessage)
        : this(exceptionType, expectedMessage, false)
    {
    }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType, string expectedMessage, bool containing)
    {
        this.ExceptionType = exceptionType;
        this.ExpectedMessage = expectedMessage;
        this.Containing = containing;
    }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType, Type resourcesType, string resourceName)
        : this(exceptionType, resourcesType, resourceName, false)
    {
    }

    public ExpectedExceptionWithMessageAttribute(Type exceptionType, Type resourcesType, string resourceName, bool containing)
    {
        this.ExceptionType = exceptionType;
        this.ExpectedMessage = ExpectedMessage;
        this.ResourcesType = resourcesType;
        this.ResourceName = resourceName;
        this.Containing = containing;
    }

    protected override void Verify(Exception e)
    {
        if (e.GetType() != this.ExceptionType)
        {
            Assert.Fail(String.Format(
                            "ExpectedExceptionWithMessageAttribute failed. Expected exception type: <{0}>. Actual exception type: <{1}>. Exception message: <{2}>",
                            this.ExceptionType.FullName,
                            e.GetType().FullName,
                            e.Message
                            )
                        );
        }

        var actualMessage = e.Message.Trim();

        var expectedMessage = this.ExpectedMessage;

        if (expectedMessage == null)
        {
            if (this.ResourcesType != null && this.ResourceName != null)
            {
                PropertyInfo resourceProperty = this.ResourcesType.GetProperty(this.ResourceName, BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public);
                if (resourceProperty != null)
                {
                    string resourceValue = null;

                    try
                    {
                        resourceValue = resourceProperty.GetMethod.Invoke(null, null) as string;
                    }
                    finally
                    {
                        if (resourceValue != null)
                        {
                            expectedMessage = resourceValue;
                        }
                        else
                        {
                            Assert.Fail("ExpectedExceptionWithMessageAttribute failed. Could not get resource value. ResourceName: <{0}> ResourcesType<{1}>.",
                            this.ResourceName,
                            this.ResourcesType.FullName);
                        }
                    }
                }
                else
                {
                    Assert.Fail("ExpectedExceptionWithMessageAttribute failed. Could not find static resource property on resources type. ResourceName: <{0}> ResourcesType<{1}>.",
                        this.ResourceName,
                        this.ResourcesType.FullName);
                }
            }
            else
            {
                Assert.Fail("ExpectedExceptionWithMessageAttribute failed. Both ResourcesType and ResourceName must be specified.");
            }
        }

        if (expectedMessage != null)
        {
            StringComparison stringComparison = this.IgnoreCase ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
            if (this.Containing)
            {
                if (actualMessage == null || actualMessage.IndexOf(expectedMessage, stringComparison) == -1)
                {
                    Assert.Fail(String.Format(
                                    "ExpectedExceptionWithMessageAttribute failed. Expected message: <{0}>. Actual message: <{1}>. Exception type: <{2}>",
                                    expectedMessage,
                                    e.Message,
                                    e.GetType().FullName
                                    )
                                );
                }
            }
            else
            {
                if (!string.Equals(expectedMessage, actualMessage, stringComparison))
                {
                    Assert.Fail(String.Format(
                                    "ExpectedExceptionWithMessageAttribute failed. Expected message to contain: <{0}>. Actual message: <{1}>. Exception type: <{2}>",
                                    expectedMessage,
                                    e.Message,
                                    e.GetType().FullName
                                    )
                                );
                }
            }
        }
    }
}

Solution 5

You can use code from this project explained in this article to create an ExpectedExceptionMessage attribute which will work with MS Test to get the desired result.

ms test (fails):

[TestClass]
public class Tests : MsTestExtensionsTestFixture
{
    [TestMethod, ExpectedExceptionMessage(typeof(System.FormatException), "blah")]
    public void Validate()
    {
        int.Parse("dfd");
    }
}

View more solutions