Test Framework Philosophy

My development team is working to implement and enforce more formal development processes than we have used in the past.  Part of this process involves deciding on which unit test framework to use going forward.  Traditionally we have used NUnit and it has worked well for our needs but now that we’re implementing Visual Studio Team System we now have MSTest available.  This has sparked a bit of a debate as to whether we should stick with NUnit or migrate to MSTest.  As we examine the capabilities of each framework and weigh each of their advantages and disadvantages I’ve come to realize that the decision is a philosophical matter.

MSTest has a bit of a bad reputation.  The general consensus seems to be that MSTest sucks.  A few weeks ago I would have thoroughly agreed with that assessment but recently I’ve come to reconsider that position.  The problem isn’t that MSTest sucks, it’s that MSTest follows a different paradigm than some other frameworks as to what a test framework should provide.

My favorite feature of NUnit is its rich, expressive syntax.  I especially like NUnit’s constraint-based assertion model.  By comparison, MSTest’s assertion model is limited, even restrictive if you’re used to the rich model offered by NUnit.  Consider the following “classic” assertions from both frameworks:

NUnit MSTest
Equality/Inequality Assert.AreEqual(e, a)
Assert.AreNotEqual(e, a)
Assert.Greater (e, a)
Assert.LessOrEqual(e, a)
Assert.AreEqual (e, a)
Assert.AreNotEqual (e, a)
Assert.IsTrue(a > e)
Assert.IsTrue(a <= e)
Boolean Values Assert.IsTrue(a)
Reference Assert.AreSame(e, a)
Assert.AreNotSame(e, a)
Assert.AreSame(e, a)
Assert.AreNotSame(e, a)
Null Assert.IsNull(a)
e – expected value
a – actual value

They’re similar aren’t they?  Each of the assertions listed are functionally equivalent but notice how the Greater and LessOrEqual assertions are handled in MSTest.  MSTest doesn’t provide assertion methods for these cases but instead relies on evaluating expressions to define the condition.  This difference above all else defines the divergence in philosophy between the two frameworks.  So why is this important?


Unit tests should be readable.  In unit tests we often break established conventions and/or violate the coding standards we use in our product code.  We sacrifice brevity in naming with Really_Long_Snake_Case_Names_So_They_Can_Be_Read_In_The_Test_Runner_By_Non_Developers.  We sacrifice DRY to keep code together.  All of these things are done in the name of readability.

The Readability Debate

Argument 1: A rich assertion model can unnecessarily complicate a suite of tests particularly when multiple developers are involved.

Rich assertion models make it possible to assert the same condition in a variety of ways resulting in a lack of consistency.  Readability naturally falls out of a week assertion model because the guess work of which form of an assertion is being used is removed.

Argument 2: With a rich model there is no guess work because assertions are literally spelled out as explicitly as they can be.
Assert.Greater(e, a) doesn’t require a mental context shift from English to parsing an expression.  The spelled out statement of intent is naturally more readable for developers and non-developers alike.

My Position

I strongly agree with argument 2.  When I’m reading code I derive as much meaning from the method name as I can before examining the arguments.  “Greater” conveys more contextual information than “IsTrue.”  When I see “IsTrue” I immediately need to ask “What’s true?” then delve into an argument which could be anything that returns a boolean value.  In any case I still need to think about what condition is supposed to be true.

NUnit takes expressiveness to another level with its constraint-based assertions.  The table below lists the same assertions as the table above when written as constraint-based assertions.

Equality/Inequality Assert.That(e, Is.EqualTo(a))
Assert.That(e, Is.Not.EqualTo(a))
Assert.That(e, Is.GreaterThan(a))
Assert.That(e, Is.LessThanOrEqualTo(a))
Boolean Values Assert.That(a, Is.True)
Assert.That(a, Is.False)
Reference Assert.That(a, Is.SameAs(e))
Assert.That(a, Is.Not.SameAs(e))
Null Assert.That(a, Is.Null)
Assert.That(a, Is.Not.Null)
e – expected value
a – actual value

Constraint-based assertions are virtually indistinguishable from English.  To me this is about as readable as code can be.

Even the frameworks with a weak assertion model provide multiple ways of accomplishing the same task.  Is it not true that Assert.AreEqual(e, a) is functionally equivalent to Assert.IsTrue(e == a)?  Is it not also true that Assert.AreNotEqual(e, a) is functionally equivalent to Assert.IsTrue(e !=a)?  Since virtually all assertions ultimately boil down to ensuring that some condition is true and throwing an exception when that condition is not true, shouldn’t weak assertion models be limited to little more than Assert.IsTrue(a)?

Clearly there are other considerations beyond readability when deciding upon a unit test framework but given that much of the power of a given framework is provided by the assertion model it’s among the most important.  To me, an expressive assertion model is just as important as the tools associated with the framework.

Your thoughts?