NUnit

NUnit With F#

Visual F#A few weeks ago I started learning F#. As I moved from the simple examples that were easily executable in F# Interactive or LINQPad though I found myself wanting to write formal unit tests to make sure I was understanding the concepts correctly. I decided to write my tests in F# for more practice and because its syntax makes it an attractive choice but in order to keep the learning curve under control a bit I thought it best to stick with a familiar test framework – NUnit. Using NUnit with F# is pretty trivial but being so new to the language I ran into a few small obstacles.

Importing Types

In order to use NUnit we need to import the types from the NUnit.Framework namespace. Importing the types is accomplished with the open keyword which looks and behaves much like the using directive in C#.

open NUnit.Framework

Don’t get too excited yet though because the compiler is probably yelling at you.

Most F# tutorials that I’ve seen provide examples that are easily run within F# Interactive (FSI), LINQPad, or a single F# Application (exe) project. They generally only include a few types and almost never span multiple files let alone multiple assemblies or languages. This is great for introducing concepts but as soon as you move to a more complex library project (like one for testing) the compiler will politely inform you of a problem depending on where you’ve placed the declaration:

Files in libraries or multiple-file applications must begin with a namespace or module declaration, e.g. ‘namespace SomeNamespace.SubNamespace’ or ‘module SomeNamespace.SomeModule’

If you’ve placed the directive at the beginning of the file before declaring a namespace or declaration you’ll see the above error. To fix this problem I like to take the namespace route because I’m just trying to group my testing types and it follows the organization pattern I’d be using were I writing the tests in C#.

namespace StringCalculatorKata.Logic.Tests

open NUnit.Framework

[]
type StringCalculatorTests() =
  []
  member x.Add_EmptyString_ReturnsZero() =
    // Test Code Omitted
  [<TestCase("1", Result=1)>]
  [<TestCase("2", Result=2)>]
  member x.Add_SingleNumber_ReturnsThatNumber calcString =
    // Test Code Omitted

Applying Attributes

With the types imported we can identify our fixtures and tests. The good news is that we accomplish this with attributes just like with other .NET languages. The bad news is that although it’s simple, it’s not immediately obvious. If you’re reading this you should already know that TestFixtureAttribute applies to the classes that hold your tests and that TestAttribute and TestCaseAttribute identify tests. But how does that translate to F#?

One answer is to define a type and decorate it with the appropriate attributes. You’ll want to make sure you’re applying the test attributes to member bindings rather than let bindings here because let bindings compile to internal members whereas member bindings compile to public members by default.

[]
type StringCalculatorTests() =
  []
  member x.Add_EmptyString_ReturnsZero() =
    // Test Code Omitted
  [<TestCase("1", Result=1)>]
  [<TestCase("2", Result=2)>]
  member x.Add_SingleNumber_ReturnsThatNumber calcString =
    // Test Code Omitted

Were you aware though that since .NET 2.0 you could apply TestFixtureAttribute to a static class as well? If you prefer this approach you can define your fixture as a module since modules compile down to static classes.

[]
module StringCalculatorTests =
  []
  let Add_EmptyString_ReturnsZero() =
    // Test Code Omitted
  [<TestCase("1", Result=1)>]
  [<TestCase("2", Result=2)>]
  let Add_SingleNumber_ReturnsThatNumber calcString =
    // Test Code Omitted

Aside from being a module definition, notice that the attributes are applied to let bindings in the module. Module level let bindings compile to public static members by default.

Assertions

So far I’ve been careful to avoid including assertions in the sample code. Not to worry, assertions in F# are the same as they’d be in any other language but you do need to be aware of a syntactic nuance that affects many of the assertion methods.

In short, the compiler’s overload resolution mechanism requires you call them using a syntactic tuple form (syntax that looks like a tuple but isn’t) rather than the curried form that’s so common in F#. This means that instead of separating the individual parameters with spaces you need to call the method like you would in C#:

type StringCalculatorTests() =
    []
    member x.Add_EmptyString_ReturnsZero() =
        let calc = new StringCalculator()
        let result = calc.Add ""
        Assert.That(0, Is.EqualTo 0)

Of course, if you’re using TestCaseAttribute with the optional Result parameter you’ll be able to eliminate the manual assertion altogether in many cases:

[]
type StringCalculatorTests() =
  [<TestCase("1", Result=1)>]
  [<TestCase("2", Result=2)>]
  member x.Add_SingleNumber_ReturnsThatNumber calcString =
    let calc = new StringCalculator()
    calc.Add calcString

Test Framework Philosophy

My development team is working to implement and enforce more formal development processes than we have used in the past.  Part of this process involves deciding on which unit test framework to use going forward.  Traditionally we have used NUnit and it has worked well for our needs but now that we’re implementing Visual Studio Team System we now have MSTest available.  This has sparked a bit of a debate as to whether we should stick with NUnit or migrate to MSTest.  As we examine the capabilities of each framework and weigh each of their advantages and disadvantages I’ve come to realize that the decision is a philosophical matter.

MSTest has a bit of a bad reputation.  The general consensus seems to be that MSTest sucks.  A few weeks ago I would have thoroughly agreed with that assessment but recently I’ve come to reconsider that position.  The problem isn’t that MSTest sucks, it’s that MSTest follows a different paradigm than some other frameworks as to what a test framework should provide.

My favorite feature of NUnit is its rich, expressive syntax.  I especially like NUnit’s constraint-based assertion model.  By comparison, MSTest’s assertion model is limited, even restrictive if you’re used to the rich model offered by NUnit.  Consider the following “classic” assertions from both frameworks:

NUnit MSTest
Equality/Inequality Assert.AreEqual(e, a)
Assert.AreNotEqual(e, a)
Assert.Greater (e, a)
Assert.LessOrEqual(e, a)
Assert.AreEqual (e, a)
Assert.AreNotEqual (e, a)
Assert.IsTrue(a > e)
Assert.IsTrue(a <= e)
Boolean Values Assert.IsTrue(a)
Assert.IsFalse(a)
Assert.IsTrue(a)
Assert.IsFalse(a)
Reference Assert.AreSame(e, a)
Assert.AreNotSame(e, a)
Assert.AreSame(e, a)
Assert.AreNotSame(e, a)
Null Assert.IsNull(a)
Assert.IsNotNull(a)
Assert.IsNull(a)
Assert.IsNotNull(a)
e – expected value
a – actual value

They’re similar aren’t they?  Each of the assertions listed are functionally equivalent but notice how the Greater and LessOrEqual assertions are handled in MSTest.  MSTest doesn’t provide assertion methods for these cases but instead relies on evaluating expressions to define the condition.  This difference above all else defines the divergence in philosophy between the two frameworks.  So why is this important?

Readability

Unit tests should be readable.  In unit tests we often break established conventions and/or violate the coding standards we use in our product code.  We sacrifice brevity in naming with Really_Long_Snake_Case_Names_So_They_Can_Be_Read_In_The_Test_Runner_By_Non_Developers.  We sacrifice DRY to keep code together.  All of these things are done in the name of readability.

The Readability Debate

Argument 1: A rich assertion model can unnecessarily complicate a suite of tests particularly when multiple developers are involved.

Rich assertion models make it possible to assert the same condition in a variety of ways resulting in a lack of consistency.  Readability naturally falls out of a week assertion model because the guess work of which form of an assertion is being used is removed.

Argument 2: With a rich model there is no guess work because assertions are literally spelled out as explicitly as they can be.
Assert.Greater(e, a) doesn’t require a mental context shift from English to parsing an expression.  The spelled out statement of intent is naturally more readable for developers and non-developers alike.

My Position

I strongly agree with argument 2.  When I’m reading code I derive as much meaning from the method name as I can before examining the arguments.  “Greater” conveys more contextual information than “IsTrue.”  When I see “IsTrue” I immediately need to ask “What’s true?” then delve into an argument which could be anything that returns a boolean value.  In any case I still need to think about what condition is supposed to be true.

NUnit takes expressiveness to another level with its constraint-based assertions.  The table below lists the same assertions as the table above when written as constraint-based assertions.

Equality/Inequality Assert.That(e, Is.EqualTo(a))
Assert.That(e, Is.Not.EqualTo(a))
Assert.That(e, Is.GreaterThan(a))
Assert.That(e, Is.LessThanOrEqualTo(a))
Boolean Values Assert.That(a, Is.True)
Assert.That(a, Is.False)
Reference Assert.That(a, Is.SameAs(e))
Assert.That(a, Is.Not.SameAs(e))
Null Assert.That(a, Is.Null)
Assert.That(a, Is.Not.Null)
e – expected value
a – actual value

Constraint-based assertions are virtually indistinguishable from English.  To me this is about as readable as code can be.

Even the frameworks with a weak assertion model provide multiple ways of accomplishing the same task.  Is it not true that Assert.AreEqual(e, a) is functionally equivalent to Assert.IsTrue(e == a)?  Is it not also true that Assert.AreNotEqual(e, a) is functionally equivalent to Assert.IsTrue(e !=a)?  Since virtually all assertions ultimately boil down to ensuring that some condition is true and throwing an exception when that condition is not true, shouldn’t weak assertion models be limited to little more than Assert.IsTrue(a)?

Clearly there are other considerations beyond readability when deciding upon a unit test framework but given that much of the power of a given framework is provided by the assertion model it’s among the most important.  To me, an expressive assertion model is just as important as the tools associated with the framework.

Your thoughts?