An experiment with automatic tests

I spent some time working with some code recently, which had some annoying habits of failing oddly in scenarios where nulls got passed into constructors. While I was trying to work around some of these issues, it struck me that tests for parameter handling for constructors are one of those annoying things that tend to make unit testing frustrating. They're annoying boiler-plate to write if you need them, and then a constructor signature changes, you end up with a lot of make-work test changes to do.

So as an exercise in "learning something new", wondered whether I could automate them in a reasonable way...

What do we want to test? url copied!

For the sake of a simple example, lets assume we want to validate that any reference parameters passed into a constructor will throw an `ArgumentNullException` if a null is supplied, and that we want to test any public constructor taking more than one parameter. Based on experience there may be some types we explicitly want to exclude from tests, but by default all public classes should be included.

Ideally these tests should also look like (and get run in the same way) as the standard unit tests for the rest of the code.

One way to achieve this... url copied!

If we want to generate unit tests that look and feel like "normal" tests, then a fairly obvious way to achieve this is to generate the source code for those tests via the T4 Template engine in Visual Studio. Reflection over the assembly we want to generate tests for will allow us to find the constructors to test, and extract data about their parameters for the purpose of generating code.

Then the code needs to generate a test for each parameter that is a reference type, where it sets one parameter to null while supplying valid defaults for the others. We expect the test to throw, so the ExpectedException attribute needs adding to each test.

Implementing all that... url copied!

The first step then, is to be able to find the set of types we want to process to make tests. That's fairly easy to achieve with reflection. You want a list of all the public classes that aren't abstract. It also seems useful to exclude any classes marked as "don't test this one". That's easy to do by tagging them with a custom attribute. That can be defined simply enough:

[AttributeUsage(AttributeTargets.Class)]
public class NoAutomaticTestAttribute : Attribute
{
}

					

Having spent some time experimenting with this code, I've come to the conclusion that (much like Razor or ASPX files) you should keep logic and code to a minimum in T4 files, so I'm trying to make as much of the logic as possible into functions that the T4 file can call. The function to fetch the relevant types can be something like:

public static IEnumerable<Type> FetchTypesToTest(Assembly assembly)
{
    return assembly.GetTypes()
        .Where(t => t.IsClass && t.IsPublic && !t.IsAbstract)
        .Where(t => t.CustomAttributes.Where(a => a.AttributeType== typeof(NoAutomaticTestAttribute)).Any() == false)
        .AsEnumerable();
}

					

Calling that code requires an assembly. For the sake of completeness that can be returned via a function too:

public static Assembly FetchAssembly(string path)
{
    return Assembly.LoadFrom(path);
}

					

That lets you generate tests for any DLL – but how do you reliably get the path to the one you want? You could hard-code it into your T4 file, but that's fragile – if someone clones your code to a different path then it can break. What may be easier is to make use of the T4 Engine's ability to ask Visual Studio to work out the values of the same path macros you can use in Post-Build Events. That requires two things. First you need to declare your template as being "aware of its hosting environment". That means setting the hostspecific attribute in the template decoration to true:

<#@ template debug="false" hostspecific="true" language="C#" #>

					

(Note that this may cause issues with hosted builds – since they don't include Visual Studio)

And secondly you need to write your template code to ask for the values of the macros:

string path = this.Host.ResolveAssemblyReference(@"$(ProjectDir)bin\$(ConfigurationName)");

					

To make use of the helper code in the test assembly, you need to make a reference to the DLL containing that code in your T4 file. For my test project that was:

<#@ assembly name="$(ProjectDir)bin\$(ConfigurationName)\AutoConstructorTest.Tests.dll" #>

					

Note that these assembly reference declarations can make use of the path macros without need for the host-awareness change above. That is only needed if you're using the macros in the code of the T4 file.

With that in place, the basic generation of test classes in the T4 file can look like:

// **
// ** Generated code - do not edit
// **
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;

namespace AutoConstructorTest.Tests
{
<#
	string path = this.Host.ResolveAssemblyReference(@"$(ProjectDir)bin\$(ConfigurationName)");
	var assembly = ConstructorTestHelper.FetchAssembly(path + @"\AutoConstructorTest.Examples.dll");
	var types = ConstructorTestHelper.FetchTypesToTest(assembly);
	foreach(var typeToTest in types)
        {
#>

	[TestClass]
	[TestCategory("AutoGenerated")]
	public class <#= typeToTest.Name #>_GeneratedTests
	{
        }
<#
        }    
#>

					

At this point I realised two things about T4 that irritate me. First is that it's really hard to format a T4 template so that the indenting looks right in both the template and in the generated output. I kind of gave up there, and accepted that the template file looks a bit odd. The second is a bit more of a challenge: this code works, but the reflection load of the target assembly puts a lock on it. That means you find yourself restarting Visual Studio a lot while you're working through writing code like this. Pretty much every time you hit build. And that gets frustrating.

An alternative here is to use a "Reflection Only Load" for the assembly you're going to process. That doesn't take the same locks, but requires slightly different code.

First up, loading the DLL to process needs adjusting to look like this:

public static Assembly FetchAssembly(string path)
{
    return Assembly.ReflectionOnlyLoadFrom(path);
}

					

Making that change may cause an error though. A reflection-only load uses a different memory scope, and dependent assemblies aren't loaded automatically. I'd put my "exclude from test" attribute into a separate assembly, and that wasn't automatically loaded into the reflection-only scope. So that needed a new initialisation method before loading the assembly to be tested:

public static void InitialiseTestHelper(string path)
{
    Assembly.ReflectionOnlyLoadFrom(path + @"\AutoConstructorTest.dll");
}

					

(Different code may need other things loaded here – it depends on what your target dll needs in memory)
And the new method needs calling in the template as well:

// **
// ** Generated code - do not edit
// **
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;

namespace AutoConstructorTest.Tests
{
<#
        string path = this.Host.ResolveAssemblyReference(@"$(ProjectDir)bin\$(ConfigurationName)");
        ConstructorTestHelper.InitialiseTestHelper(path);
        var assembly = ConstructorTestHelper.FetchAssembly(path + @"\AutoConstructorTest.Examples.dll");
        var types = ConstructorTestHelper.FetchTypesToTest(assembly);
        foreach(var typeToTest in types)
        {
#>

        [TestClass]
        [TestCategory("AutoGenerated")]
        public class <#= typeToTest.Name #>_GeneratedTests
        {
        }
<#
        }    
#>

					

This code worked, and allowed the build and generation to run without locking the target assembly every time. But it raised a new issue. The code for filtering out assemblies with the "don't test this" attribute stopped working, and it returned all types, whether they included the custom attribute or not.

A bit of head scratching about that lead me to two things. First, attaching the debugger to the T4 Template still locks the target DLL. So you probably still need to re-start Visual Studio after you've done that and before you do a rebuild of your code. But the other thing was subtler. The test for "does the class being examined have the custom attribute" is comparing the type data for the attribute found on the class to typeof(NoAutomaticTestAttribute). A type loaded into a reflection-only scope is not the same thing as a type loaded into the normal execution scope. So that test can't work any more. There may well be a better way to address this, but the easiest change to make is to compare the names of the types, rather than their type objects directly:

            var x = assembly.GetTypes()
                .Where(t => t.IsClass && t.IsPublic && !t.IsAbstract)
                .Where(t => t.CustomAttributes.Where(a => a.AttributeType.Name == typeof(NoAutomaticTestAttribute).Name).Any() == false)
                .AsEnumerable();

					

With that change made, the code worked again.

Next step then, is to generate a test for each of the constructor parameters we want to check. Given a type, we can extract the set of constructors easily enough. We just want any public constructors for instances of the type:

public static IEnumerable<ConstructorInfo> FetchConstructorsToTest(Type type)
{
    return type
        .GetConstructors(BindingFlags.Public | BindingFlags.Instance)
        .AsEnumerable();
}

					

The template can then iterate all the constructors, and for each one iterate all of the constructor parameters, ignoring any ones which take value types. (As they won't accept null) And for each of these it can generate variables for the set of parameters and a call to the class we want to test:

    [TestClass]
    [TestCategory("AutoGenerated")]
    public class <#= typeToTest.Name #>_GeneratedTests
    {

<#
        var constructors = ConstructorTestHelper.FetchConstructorsToTest(typeToTest);
        foreach(var constructor in constructors)
        {
            var parameters = constructor.GetParameters();
            foreach(var nullParameter in parameters)
            {
                if(nullParameter.ParameterType.IsValueType) continue;
#>
        [TestMethod]
        [ExpectedException(typeof(ArgumentNullException))]
        public void <#= typeToTest.Name #>_Constructor_<#= parameters.Count() #>Params_<#= nullParameter.Name #>_IsNull_Throws()
        {
<#
                foreach(var parameter in parameters)
                {
                    if(parameter.Name == nullParameter.Name)
                    {
#>
                <#= ConstructorTestHelper.GenerateTypeName(parameter.ParameterType) #> <#= parameter.Name #> = null;
<#
                    }
                    else
                    {
#>
                var <#= parameter.Name #> = <#= ConstructorTestHelper.FetchDefaultValue(parameter) #>;
<#
                    }
                }
#>

                var sut = new <#= typeToTest.FullName #>(<#= ConstructorTestHelper.GenerateConstructorParameterList(parameters) #>);
        }

<#
            }
        }
#>
    }
<#
    }

					

That looks a bit complicated, but that's mostly down to mixing the T4 logic with its output. For each constructor found, it grabs the set of parameters. It iterates these and generates a TestMethod for each one that is not a value type. These get named based on what they represent - "a test that nulling the nth parameter throws an exception". Inside that test the code iterates the set of parameters again. For each one it checks if this is the parameter we're going to set to null. If it is, it outputs a variable with the right type and a null value. If not, it calls a function to fetch a sensible default value for the variable. Finally it generates a call to the constructor of the type being tested, passing in the variables it has created.

The constructor parameters can be generated very simply – by concatenating a list of parameter names:

public static string GenerateConstructorParameterList(IEnumerable<ParameterInfo> parameters)
{
    return String.Join(", ", parameters.Select(p => p.Name));
}

					

You also need the names of the types you're creating. The simplest approach to that is just to use parameter.FullName. But that only works for simple types. If you try that with something generic you'll end up code that doesn't compile. The .Net formal name for a generic type uses back-ticks numbers and square brackets to describe it. Things a bit like List1[Int32]` – and that's not valid C#. There may be a better way of handling this, but my first stab was to generate the C#-friendly name from the type data like so:

public static string GenerateTypeName(Type t)
{
    if (t.IsGenericType)
    {
        string name = t.FullName.Substring(0, t.FullName.IndexOf('`'));

        return name + "<" + String.Join(", ", t.GetGenericArguments().Select(a => GenerateTypeName(a))) + ">";
    }
    else
    {
        return t.FullName;
    }
}

					

That handles both cases.

Generating the default values for the constructor parameters needed a bit more thought. For the purposes of my experiment I've only addressed strings, simple value types like ints and the reference types the tests really care about. But code for other types wouldn't be too hard to add if you needed to.

Any string paramters can take the value string.Empty. Anything that's a simple value type can be initialised to a sensible default with the runtime's <a href="https://msdn.microsoft.com/en-us/library/system.activator.createinstance(v=vs.110).aspx" target="_blank" rel="noopener noreferrer">Activator.CreateInstance()</a> method. And whatever's left is a reference type that we can create a test mock for with new Mock(). (I'm using the moq framework here – but other mocking tools would work as well)

public static string FetchDefaultValue(ParameterInfo parameter)
{
    string defaultValue = null;
    if (parameter.ParameterType.Name == "String")
    {
        defaultValue = "String.Empty";
    }
    else if (parameter.ParameterType.IsValueType)
    {
        var v = Activator.CreateInstance(parameter.ParameterType);
        defaultValue = v.ToString().ToLower();
    }
    else
    {
        defaultValue = "new Mock<" + GenerateTypeName(parameter.ParameterType) + ">().Object";
    }

    return defaultValue;
}

					

With that in place, you can generate tests for quite a wide spread of classes. They come out looking like this:

[TestClass]
[TestCategory("AutoGenerated")]
public class ValidClass2_GeneratedTests
{

    [TestMethod]
    [ExpectedException(typeof(ArgumentNullException))]
    public void ValidClass2_Constructor_2Params_s1_IsNull_Throws()
    {
            System.String s1 = null;
            var i = new Mock<AutoConstructorTest.Examples.ISomeInterface>().Object;

            var sut = new AutoConstructorTest.Examples.ValidClass2(s1, i);
    }

    [TestMethod]
    [ExpectedException(typeof(ArgumentNullException))]
    public void ValidClass2_Constructor_2Params_i_IsNull_Throws()
    {
            var s1 = String.Empty;
            AutoConstructorTest.Examples.ISomeInterface i = null;

            var sut = new AutoConstructorTest.Examples.ValidClass2(s1, i);
    }

    [TestMethod]
    [ExpectedException(typeof(ArgumentNullException))]
    public void ValidClass2_Constructor_3Params_s1_IsNull_Throws()
    {
            System.String s1 = null;
            var s2 = String.Empty;
            var i = new Mock<AutoConstructorTest.Examples.ISomeInterface>().Object;

            var sut = new AutoConstructorTest.Examples.ValidClass2(s1, s2, i);
    }

    [TestMethod]
    [ExpectedException(typeof(ArgumentNullException))]
    public void ValidClass2_Constructor_3Params_s2_IsNull_Throws()
    {
            var s1 = String.Empty;
            System.String s2 = null;
            var i = new Mock<AutoConstructorTest.Examples.ISomeInterface>().Object;

            var sut = new AutoConstructorTest.Examples.ValidClass2(s1, s2, i);
    }

    [TestMethod]
    [ExpectedException(typeof(ArgumentNullException))]
    public void ValidClass2_Constructor_3Params_i_IsNull_Throws()
    {
            var s1 = String.Empty;
            var s2 = String.Empty;
            AutoConstructorTest.Examples.ISomeInterface i = null;

            var sut = new AutoConstructorTest.Examples.ValidClass2(s1, s2, i);
    }

}

					

And the Visual Studio test explorer can run them:

Conclusions url copied!

Well it works – you can indeed generate these boilerplate tests.

If you want a working example, the solution that the snippets above are taken from is available on GitHub for you to play with.

But it would be fair to say that the T4 templating experience isn't that fantastic while you write stuff like this. You can make it a bit better by installing an add-on like the "Tangible T4 Editor" to get some syntax highlighting and other editor features. However the issues around assembly locking still cause a lot of "click build, wait for the compiler to time out while trying to overwrite a locked assembly, grumble, close Visual Studio, open it again and re-try the build" issues. (Which is as annoying to experience as it is to type out...)

There's also the problem that if you get something wrong in your T4 template it can generate bad C#. That can be a subtle issue, say missing a semi-colon or a brace – which can be quite hard to spot with the noise of code and data in a template. Or it can be a bigger issue. Sometimes the T4 engine will generate half a file and stop on an error – leaving a wholly invalid fragment of C# in the file. Or if the error prevents the engine doing any templating you get a file that just contains ErrorGeneratingOutput. All of these will prevent the build of your solution until you tidy them up. But if the T4 error is caused by an issue in your ordinary C# code (like messing up one of the helper functions) then you're stuck in a vicious circle of not being able to compile your helper because the T4 generated code is broken because of the helper.... etc.... etc....

Either that means I'm a bit mad for trying to factor out the logic into more readable (and testable) code, or it means that you really need to put that stuff into a separate DLL to allow it to be compiled individually. I'm musing on the idea that maybe the helper functions should just be declared at the top of the T4 file.

Doing this has taught me some interesting stuff, but were I to address this sort if issue again then I'll probably try googling for a pre-existing library for automatic test generation, and ignore the desire to have these tests look like my other ones.

Not Invented Here Syndrome can help you learn – but it's still not a good thing...

↑ Back to top