Listen to what your tests are telling you...

One of the things that the masters of Unit Testing say is that tests should always be repeatable and deterministic. No matter when, or how many times you run a test, if the System Under Test stays the same, the results of the test should stay the same. So I found myself a bit confused recently when some tests went from reliable to unreliable results. Turns out the tests were right, and they were pointing out a silly bug I'd introduced by accident...

The scenario

The code had started off with a class to store some state. To make this clearer, I'll strip it back to the minimum code – just a property to store a value:

public class OriginalClass
{
    public int Value { get; set; }
}

					

That class had a couple of tests which created an object, fiddled with some values and asserted that the values ended up right. Again, stripping it back to the basics:

[TestMethod]
public void FirstTest()
{
    var sut = new OriginalClass();
    sut.Value += 1;

    Assert.AreEqual(1, sut.Value);
}

[TestMethod]
public void SecondTest()
{
    var sut = new OriginalClass();
    sut.Value += 2;

    Assert.AreEqual(2, sut.Value);
}

					

Unsurprisingly those tests pass:

Over the course of a few further cycles of "write a test, change some code, run the tests" I found myself realising that the class at the centre of this issue could be used more simply with some default state. That lead me to add a static "Default" to the class and revise the tests a bit:

[TestClass]
public class IncorrectClassTests
{
    [TestMethod]
    public void FirstTest()
    {
        var sut = IncorrectClass.Default;
        sut.Value += 1;

        Assert.AreEqual(1, sut.Value);
    }

    [TestMethod]
    public void SecondTest()
    {
        var sut = IncorrectClass.Default;
        sut.Value += 2;

        Assert.AreEqual(2, sut.Value);
    }
}

					

But at this point I noticed that the tests became unreliable. Clicking "run all tests" lead to:

But running only the failed tests succeeded:

I scratched my head and re-ran those tests a few times, but the results were consistently inconsistent. What could cause that to happen?

Finding the mistake...

The code I was working on was using assorted other libraries, so I got a bit distracted by trying to work out whether I was using one of them incorrectly. But the problem was somewhat closer to home.

The key problem was that change to include a default state of the class. I suspect you can probably spot the error:

public class IncorrectClass
{
    public static readonly IncorrectClass Default = new IncorrectClass() { /* some initialisation */ };

    public int Value { get; set; } 
}

					

My silly mistake here is that because IncorrectClass is a class, it's accessed by reference. And since it's also a static, that means that all of the code is referring to the same instance of IncorrectClass. Hence if you run a single test the object is created, and passes the test and then destroyed. But if you run multiple tests, the object is created and reused. So the initial state is wrong for the second (and subsequent) tests...

A bit of a facepalm moment there. What I should have used was more like:

public class CorrectClass
{
    public static CorrectClass Default { get { return new CorrectClass() { /* some initialisation */ }; } }

    public int Value { get; set; }
}

					

That ensures that the "default" is created with the correct state each time.

The important conclusion:

What this really proves is that even when the test results seem odd, they can still be pointing out that it's actually your code that's wrong...

↑ Back to top