AutoFixture Default Builders

AutoFixture takes the heavy lifting out of creating test fixtures. If you haven’t used it before and want an intro, then check out the wiki.

I’ve experienced it’s use on a number of projects, and would generally recommend it.

However, some care must be taken as it generates some of the types in a constrained non-deterministic way. So if applied to non-anonymous variables that affect the outcome of the test then the test may fail intermittently.

AutoFixture comes with a set of default builders encapsulating the rules for creating a variety of .NET CTS types. It also provides points of extensibility that allow you to customize the rules.

The library has been developed over a number of years, and the documentation on the rules is quite fragmented. So I thought I’d dig into the source code and document what I find, as much for my reference and benefit as yours.

Note. The following applies to the current version (v3.0) of AutoFixture.

Numbers

var fixture = new Ploeh.AutoFixture.Fixture();

Console.WriteLine("{0}, {1}, {2}",
                  fixture.Create<int>(),
                  fixture.Create<double>(),
                  fixture.Create<byte>());
79, 236, 44

The default builder for numbers is RandomNumericSequenceGenerator. It generates a sequence of unique numbers for the following types – Byte, Decimal, Double, Int16, Int32, Int64, SByte, Single, UInt16, UInt32, UInt64.

Unique numbers are generated randomly from the set [1, 255]. Once these are used up they are then be generated from the set [256, 65,535]. And finally from the set [65,536,  2,147,483,647]. When all numbers within the final set have been used AutoFixture will start again from the first set.

fixture.Customizations.Add(
    new RandomNumericSequenceGenerator(1, 10)
);

Console.WriteLine("{0}, {1}, {2}",
                  fixture.Create<int>(),
                  fixture.Create<int>(),
                  fixture.Create<int>());
10, 6, 8

RandomNumericSequenceGenerator has a constructor that takes a variable number of Int64 limits used to constrain the sets. There must be at least 2 limits, hence one set, and they must be specified in numeric order.

By adding an instance of RandomNumericSequenceGenerator to the Fixture.Customizations collection, you can constrain the generated numbers by specifying custom limits; as shown above where the numbers are constrained between 1 and 10.

fixture.Customizations.Add(
    new NumericSequenceGenerator()
);

Console.WriteLine("{0}, {1}, {2}",
                  fixture.Create<int>(),
                  fixture.Create<int>(),
                  fixture.Create<int>());
1, 2, 3

The RandomNumericSequenceGenerator was introduced with v3.0. Earlier versions used the NumericSequenceGenerator which generates numbers in sequence starting at 1. If you wish to have AutoFixture generate numbers in a deterministic manner then you can specify the usage of NumericSequenceGenerator by adding it to the Customizations collection.

Chars

Console.WriteLine("{0}, {1}, {2}",
                  fixture.Create<char>(),
                  fixture.Create<char>(),
                  fixture.Create<char>());
Y, /, A

The default builder for char is RandomCharSequenceGenerator, which generates random characters from the printable ASCII character set (“!” (33) to “~” (126)).

fixture.Customizations.Add(
    new CharSequenceGenerator()
);

Console.WriteLine("{0}, {1}, {2}",
                  fixture.Create<char>(),
                  fixture.Create<char>(),
                  fixture.Create<char>());
!, ", #

For a more deterministic approach the CharSequenceGenerator builder can be used to cycle through the printable ASCII character set.

Strings

Console.WriteLine("{0}", fixture.Create<string>());
 eee9e1b5-70a1-4b61-952d-c6fa4c14b166

StringGenerator is a builder that has a constructor accepting a delegate (Func<object>), which is invoked in order to generate strings.

StringGenerator is registered as the default builder for strings an initialized with the following function – () => Guid.NewGuid(). It therefore generates unique Guids for strings.

Console.WriteLine("{0}", fixture.Create<string>().Substring(0, 10));
22ca485a-7

A Guid, including the hythens, is 36 characters in length. If you require a string with less characters, and do not want to go to the lengths of creating and registering a custom string builder, then the easiest thing to do is apply the Substring method to the resultant string.

Console.WriteLine("{0}", string.Join(string.Empty, fixture.CreateMany<string>(5)));
edd22091-e048-4d22-a41c-0c50482f65d96d741962-9b9b-4431-9d17-6993986933cd1a3814e5-3989-4448-8ea2-5bc8de4d55744314904f-8884-4af4-8253-45c7f1dc11306b0e2212-bae9-44de-8754-5dd62ebcf2a0

Generating a longer string is a little more complex, but can be achieved by using the Fixture.CreateMany() method to generate a collection of strings, and then join these together using String.Join.

using System.ComponentModel.DataAnnotations;
public class Product
{
    public int ProductId { get; set; }
   
    [StringLength(10)]
    public string Name { get; set; }
}

Console.WriteLine("{0}", fixture.Create<Product>().Name);
60ebc6bc-5

ConstrainedStringGenerator is registered as a default handler for strings, in addition to StringGenerator. The difference is that ConstrainedStringGenerator is used to constrain strings when they are annotated with the StringLength  attribute. It generates strings by concatenating Guids, and then taking a Substring of the Guids to ensure an exact length.

fixture.Register<string>(() => "Lorem ipsum dolor sit amet, consectetur adipiscing elit.");

Console.WriteLine("{0}", fixture.Create<string>());Console.WriteLine("{0}", fixture.Create<string>());
Lorem ipsum dolor sit amet, consectetur adipiscing elit.

AutoFixture has a Register method, which allows the function used to generate instances of a specific type to be overridden. In the above example, the Register method overrides the default method used for generating strings to one that generates Lorem Ipsum text. not sure how useful this is; it’s just an example.

One small caveat with overriding the string function as shown above. If you the try to generate a Uri, you will receive the following error ” Invalid URI: The hostname could not be parsed.”. the Uri utilizes the string function to generate the hostname, and hostnames can only comprise of a limited number of characters, which does not include spaces or commas.

DateTime

Console.WriteLine("{0}, {1}, {2}",
                  fixture.Create<DateTime>(),
                  fixture.Create<DateTime>(),
                  fixture.Create<DateTime>());
04/11/2016 00:59:14, 27/11/2017 10:33:06, 26/12/2015 04:33:45

RandomDateTimeSequenceGenerator is the default builder for DateTime objects. It has a constructor that takes a minimum and maximum DateTime, and it randomly generates a date between these limits. by default the limits are set to DateTime.Now.AddYears(-2) and DateTime.Now.AddYears(2).

fixture.Customizations.Add(
    new RandomDateTimeSequenceGenerator(new DateTime(2017, 1, 1), new DateTime(2017, 1, 31))
);

Console.WriteLine("{0}, {1}, {2}",
        fixture.Create<DateTime>(),
        fixture.Create<DateTime>(),
        fixture.Create<DateTime>());
06/01/2017 07:11:33, 26/01/2017 19:17:35, 01/01/2017 16:18:43

By adding the RandomDateTimeSequenceGenerator to the Customizations collection and specifying minimum and maximum DateTime limits you can customize the range from which DateTime values are generated. In the example above values are generated for the year 2017.

AutoFixture has 2 addition builders for DateTime: StrictlyMonotonicallyIncreasingDateTimeGenerator, which takes a seed DateTime as a constructor parameter, and increments each subsequent generated DateTime by one day; and CurrentDateTimeGenerator which returns DateTime.Now.

Uri

Console.WriteLine("{0}", fixture.Create<Uri>());

fixture.Inject(new UriScheme("ftp"));
Console.WriteLine("{0}", fixture.Create<Uri>());
http://0d171f49-5624-417f-a826-2866b166e225/
ftp://cf56a59a-5390-4c78-807a-7a4cd01b6eec/

UriGenerator is used to build instances of Uri, defaulting the scheme to “http” and the authority generated by StringGenerator, hence a Guid.

The scheme can be customized by injecting an instance of UriScheme with the scheme passed to the constructor, as shown above for “ftp”.

The authority can be customized by registering a new string builder function. But as mentioned in the Strings section, the string generator function should only generate strings containing only characters compatible with URIs.

Regular Expressions

public class Contact
{
    [RegularExpression(@"^[2-9]\d{2}-\d{3}-\d{4}$")]
    public string Telephone { get; set; }
}

Console.WriteLine("{0}", fixture.Create<Contact>().Telephone);
601-000-1101

As mentioned in the Strings section, the ConstrainedStringGenerator builds strings of specific lengths where AutoFixture is used to generate a custom type that has a string property annotated with the StringLength attribute.

Similarly, the RegularExpressionGenerator builds strings annotated with a RegularExpression attribute.

Booleans

Console.WriteLine("{0}, {1}",
                  fixture.Create<bool>(),
                  fixture.Create<bool>());
True, False

BooleanSwitch is the default builder for Booleans, generating in a deterministic manner, alternating between True and False.

Guids

Console.WriteLine("{0}", fixture.Create<Guid>());
2fac899e-55a2-4161-a90f-ec7e7b87ca92

GuidGenerator returns Guid.NewGuid().

Delegates

public bool void Process(int orderId);

Console.WriteLine("{0}, {1}", func.Method, func.Invoke(1));
Boolean lambda_method(System.Runtime.CompilerServices.Closure, Int32), True

DelegateGenerator is the default builder for delegates. It returns a delegate pointing to a dynamically generated method.

If the delegate has a return value, then the method will return a generated instance of that value when invoked.

MailAddress

Console.WriteLine("{0}", fixture.Create<MailAddress>());
"43b37eb4-48ee-49e2-bc8c-1bed70fc1796" <43b37eb4-48ee-49e2-bc8c-1bed70fc1796@example.com>

The MailAddressGenerator builder generates instances of MailAddress.

The user part of the address is generated using the registered string generator, which by default returns Guid.NewGuid(). As mentioned in the Url section, be careful when registering a custom string generator function, as it must only return compatible characters, so no spaces in commas for example.

The host part of the address is generated using the DomainNameGenerator builder, which randomly returns one of the following domains – example.com, example.net, example.org.

fixture.Register(() => new DomainName("acmecorp.com"));

Console.WriteLine("{0}", fixture.Create());

"393afa04-be57-4275-afbf-7349a1ba54d3" <393afa04-be57-4275-afbf-7349a1ba54d3@acmecorp.com>

If you want to use a custom host, then you can register a custom function for generating DomainName instances, as shown above.

AutoFixture

There are more ways than one to skin a cat, or if you are a late 19th century west country gentleman, there are more ways of killing a cat than choking it with cream.

And similarly there are more ways than one to unit test an application. Martin Fowler, who is much more intelligent and experienced than I, discusses this in his blog here. In particular he talks about 2 schools of thought with regard to collaborator isolation.  Classic-style testers are happy to test their units in collaboration with their dependencies, and will only isolate a unit from a dependency if need be.  Whereas Mockist-style testers use test doubles, and in particular mocks, to isolate their units from all dependencies. Jay Fields introduced the terminology, which I like very much, for these schools of thought as sociable testing versus solitary testing.

Like flavours of ice-cream, there’s no right or wrong. Martin classes himself as a classic-style tester, so it certainly can’t be wrong. However, if the community blogs and forums are a good indicator, I would say that the majority of testers are mockists. Or at least, the mockist-style is the current flavour of the month.

The aim of unit testing is to verify the behaviour of a unit, or SUT (System Under Test), ensuring that all relevant paths – happy, alternative, and error – are adequately covered. Mockists will typically isolate the SUT from it’s dependencies by creating test doubles using a mocking framework such as Moq, NSubstitute, or FakeItEasy.

Mocking frameworks greatly simplify unit testing by taking a lot of the leg work out of creating objects that simulate the behaviour of the dependencies. Mocking frameworks save a lot of lines of code, and additionally provide functionality to help verify the communication between the SUT and the dependency. But I’m not here to talk about mocking. I’m here to talk about how we can further simplify unit testing and save even more lines of code by using AutoFixture to help us construct inputs, or test fixtures, for our SUT.

AutoFixture is an open source library created by Mark Seemann. To quote the blurb on the AutoFixture GitHub repository –

“AutoFixture is an open source library for .NET designed to minimize the ‘Arrange’ phase of your unit tests in order to maximise maintainability.Its primary goal is to allow developers to focus on what is being tested rather than how to setup the test scenario, by making it easier to create object graphs containing test data.”

It takes the heavy lifting out of creating test fixtures, reducing the coding effort and lines of code required to create and maintain tests.

My SUT is FilmService, as shown below, and I want to test the happy path of the AddFilm method.

using Sandbox461.Models;
using System;

namespace Sandbox461.Domain
{
    public class FilmService
    {
        private Sandbox461DbContext db;

        public FilmService(Sandbox461DbContext db)
        {
            this.db = db;
        }

        public Film AddFilm(Film film)
        {
            if (film == null)
            {
                throw new ArgumentNullException();
            }

            db.Films.Add(film);
            db.SaveChanges();

            return film;
        }
    }
}

As mockists the first thing we do is create mocks for all classes that our SUT depends on. In this case, an instance of Sandbox461DbContext is injected into FilmService via the constructor. FilmService therefore depends on Sandbox461DbContext.

The happy path of the AddFilm method collaborates with Sandbox461DbContext in 2 ways. Firstly, the Add method of the Films property – which is of type DbSet<Film> – is called. And secondly, the SaveChanges method of Sandbox461DbContext is called. In order to verify that the SUT calls both these methods, we need to create mocks for both the Sandbox461DbContext class and the Films property, and we need to set up the mock for Sandbox461DbContext to return the mock for the Films property.

We then create an instance of FilmService and call the AddFilm method. However, the AddFilm method requires an instance of Film, and in this case we manually create and that instance and set the required properties to appropriate values. The AddFilm method is then called.

The test verifies that the Add method of the Films mock is called once – hence the instance of film is added to the Films set. It then verifies that the SaveChanges method of the Sandbox461DbContext mock is called once. And finally the test asserts that the AddFilm method returns the Index view.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;
using System.Data.Entity;
using Sandbox461.Models;
using Sandbox461.Domain;
using System.Collections.Generic;

namespace Sandbox461Tests.Domain
{
    [TestClass]
    public class FilmServerTests
    {
        [TestMethod]
        public void AddFilm_HappyPath()
        {
            // Arrange
            var mockFilmsSet = new Mock<DbSet<Film>>();
            var mockContext = new Mock<Sandbox461DbContext>();
            mockContext.Setup(m => m.Films).Returns(mockFilmsSet.Object);

            var sut= new FilmService(mockContext.Object);

            var film = new Film
            {
                FilmId = 0,
                Title = "Test Title",
                Actors = new List<FilmMaker>()
                {
                    new FilmMaker() { FilmMakerId = 1, FirstName = "Billy Bob", LastName = "Thornton" },
                    new FilmMaker() { FilmMakerId = 1, FirstName = "Martin", LastName = "Freeman" },
                    new FilmMaker() { FilmMakerId = 1, FirstName = "Ross", LastName = "Whitehead" }
                },
                Directors = new List<FilmMaker>()
                {
                    new FilmMaker() { FilmMakerId = 1, FirstName = "Randall", LastName = "Einhorn" },
                    new FilmMaker() { FilmMakerId = 1, FirstName = "Michael", LastName = "Uppendahl" },
                    new FilmMaker() { FilmMakerId = 1, FirstName = "Adam", LastName = "Bernstein" }
                }
            };

            // Act
            var result = sut.AddFilm(film);

            // Assert
            mockFilmsSet.Verify(m => m.Add(It.IsAny<Film>()), Times.Once());
            mockContext.Verify(m => m.SaveChanges(), Times.Once());

            Assert.IsInstanceOfType(result, typeof(Film));
        }
    }
}

The instance of Film takes quite a bit of effort, and quite a few line of code to set-up. And as we develop the application Film will most likely be extended to include addition properties, such as StoryLine, Genres, Certificate, and Budget, thus becoming quite complex, and requiring many line of code to manually create the instance.

You may be questioning why I created 3 instances of FilmMaker in each of the Actors and Directors collections, where they are not required for this test. The reason is that when I introduce AutoFixture to the test, then this is exactly what AutoFixture will generate. So I wanted a like-for-like comparison.

Another issue with manually creating our test fixtures is that every time we modify the classes, possibly adding required properties, then we have to modify our tests. It would be nice if we could avoid this overhead.

So how can we reduce the effort required to set-up and maintain our test fixtures? This is cue for AutoFixture to enter stage left..

I’ve refactored the test to use AutoFixture, by replacing the explicit creation of the Film object with the following 2 lines of code –

var fixture = new Fixture();
var film = fixture.Create<Film>();

The first line of code creates an instance of the Ploeh.AutoFixture.Fixture class. And the second line uses the Create method of the Fixture instance to generate a “test fixture” of type Film.

If we take a look at the object graph for film, as shown below, we can observe the following –

  • AutoFixture populates all publically set-able fields.
  • Any fields that have private setters are not populated.
  • Unless they are set via the constructor, is which case AutoFixture will pass appropriate values through the constructor.
  • AutoFixture uses Constrained Non-Determinism to populate the values. Basically, it uses algorithms to generate non-deterministic values that are likely to stay away from any of our tests boundary conditions. The reason Mark states for using non-deterministic values is that it enforces good discipline by making it impossible to hard-code expected values in our tests. The algorithms can however be customized if need be.
  • For strings, the algorithm generates a new GUID converted to a string.
  • Numbers are random, and for integers the algorithm generates random numbers within the range of [1, 255] for the first 255 numbers, and then [256, 32767] for the remaining.
  • DataTimes are random, constrained to the past 2 and next 2 years.
  • Chars are also random.
  • For collections, such as List<T>, AutoFixture will populate the collection with 3 entries, because 3 is many. Although the number of entries can be changed by setting the Fixture.RepeatCount property.

film

The revised test is as follows –

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;
using System.Data.Entity;
using Sandbox461.Models;
using Sandbox461.Domain;
using Ploeh.AutoFixture;
using System.Collections.Generic;
using Ploeh.AutoFixture.AutoMoq;

namespace Sandbox461Tests.Domain
{
    [TestClass]
    public class FilmServerTests
    {
        [TestMethod]
        public void AddFilm_HappyPath_WithAutoFixture()
        {
            // Arrange
            var mockFilmsSet = new Mock<DbSet<Film>>();
            var mockContext = new Mock<Sandbox461DbContext>();
            mockContext.Setup(m => m.Films).Returns(mockFilmsSet.Object);

            var sut= new FilmService(mockContext.Object);

            var fixture = new Fixture();
            var film = fixture.Create<Film>();

            // Act
            var result = sut.AddFilm(film);

            // Assert
            mockFilmsSet.Verify(m => m.Add(It.IsAny<Film>()), Times.Once());
            mockContext.Verify(m => m.SaveChanges(), Times.Once());

            Assert.IsInstanceOfType(result, typeof(Film));
        }
    }
}

In summary, if you’re a mockist then you want to mock the dependencies, create test fixtures for the inputs, and then verify the behaviour and outputs for all paths in your SUT. A mocking framework will simplify the mocking of dependencies, whilst AutoFixture will simplify the creation of test fixtures. You will write less code, and your code will be durable as you will most likely not have to modify test code based on modification to the test fixture classes.

For more information on AutoFixture check out Mark Seemann’s blog. And consider buying him a cup of coffee. It’s a worthy read.

Also, check out the AutoFixture GitHub repository.

And install AutoFixture from Nuget.

 

 

 

 

Self testing code != TDD != Unit testing

Test Driven Development (TDD) has been a buzz-word in the industry for many years now, and with the increasing popularity of frameworks that support isolation of components – and I’m thinking MVC here – TDD is a skill-set that is very much in demand. But is this simply because TDD is being equated with self-testing code, and developers are not standing back and questioning whether self-testing code could be produced in a more productive and engaging way?

Of course I would not consider a work item to be complete unless it has, or has been considered for, a good coverage of workable tests. But I feel a little uneasy with adopting a “always test-first” approach to development. Sometimes it may be beneficial to break away from this approach in order to go with the flow of development. Also, I feel a little uneasy with the prevalent view that you should be able to fully test all units of code in isolation from dependencies, through the use of interfaces and mocking. Testing can be simplified in some circumstances by including dependencies. The number of tests can be reduced, and more effort can be spent on writing good well-factored code rather than on mocking complex objects. I also feel uncomfortable with a drive for 100% test coverage. Does this become an impediment to change?

Don’t get me wrong, TDD is a good approach to development. Just not in all situations. And it’s not the only approach.

The reason I am writing this post is that I’ve just watched a series of Google hangout videos on YouTube – called “Is TDD dead?” – where Martin Fowler, Kent Beck, and David Hannson debate these issues. They are well worth a look –

My take aways from the discussions are as follows –

1) TDD is an approach which produces self-testing code. But self-testing code can also be produced without following the TDD approach. It is not necessary to test-first. Test-after can produce the same test coverage. Self-testing code != TDD.

2) There is a drive in the development community to fully isolate all layers and units of code so that they can be tested independently from other layers and units. Code is therefore designed with unit-testing in mind, and as such may not be optimally designed for comprehension. This drive is a consequence of the uptake in the TDD approach. But TDD equates to red-green-refactor. It does not require the full isolation of units of code, and in many cases the approach can be followed without doing so, as long as the tests are designed to be repeatable. TDD != Unit Testing. But loose coupling of code is not necessarily a bad thing. There are other benefits besides testing. Such as the ability to switch dependencies such as data providers and services.

3) A rigid approach to TDD may lead to over-testing. What is good test coverage? Enough so that you can confidently change the code with the expectation that any errors you introduce will be highlighted by a failed test. But not so much that more effort is spent changing and maintaining tests then on changing code.

4) the TDD approach or discipline is a good one to have, but should not necessarily be followed in all situations. Whether it is used or not, the goal should be to implement a good set of regression tests.