Showing posts with label Parameterized Unit Tests. Show all posts
Showing posts with label Parameterized Unit Tests. Show all posts

Thursday, May 2, 2013

Parameterized Test Patterns using Microsoft Pex

We talked before about the difference between unit tests and parameterized unit tests. In this post we will talk about common patterns for writing good parameterized unit tests. Keep in mind that we will use these tests with Microsoft Pex (as an automatic test input generation tool) to get test inputs that trigger all the possible scenarios of the code-under-test.

Before anything, let’s clarify what are the questions we want to answer using the parameterized unit tests. There are  two core questions:

  • What are good scenarios (sequences of method calls) to exercise the code-under-test? (Coverage)
  • What are good assertions that can be stated easily without re-implementing the algorithm? (Verification)

A parameterized unit test is only useful if it provides answers for both questions:

  • Without sufficient coverage, i.e. if the scenario is too narrow to reach all the code-under-test, the extent of the test is limited.
  • Without sufficient verification of the computed results, i.e. if the test does not contain enough assertions, the test does not check that the code is doing the right thing. All it would check for is that the code-under-test does not crash.

Test Patterns

1- Arrange, Act, Assert

The ’AAA’ (Triple-A) is a well-known pattern for writing unit tests. It applies to parameterized unit tests as well. A parameterized unit test using this patter is organized in three sections:

  • Arrange: set up the unit under test
  • Act: exercise the unit under test, capturing any resulting state
  • Assert: verify the behavior through assertions

An example of this pattern in a traditional unit test:


[Test]
pubic void AddItem()
{
// arrange
var list = new ArrayList();
var item = new object();
// act
list.Add(item);
// assert
Assert.IsTrue(list.Count == 1);
}


An example of this pattern in a parameterized unit test:




[PexMethod]
pubic void AddItem(object item)
{
// arrange
var list = new ArrayList();
// act
list.Add(item);
// assert
Assert.IsTrue(list.Count == 1);
}


2- Assume, Arrange, Act, Assert


This pattern is an extension of the first pattern where an Assumption section is added at the beginning. An assumption restricts possible test inputs, acting as a filter. A parameterized unit test using this pattern is organized in four sections:



  • Assume: assume preconditions over the test inputs

  • Arrange: set up the unit under test

  • Act: exercise the unit under test, capturing any resulting state

  • Assert: verify the behavior through assertions


The following example tests that adding an element to any list increments the Count property. We use an assumption to filter out the case where list is a null reference.




[PexMethod]
void AssumeActAssert(ArrayList list, object item)
{
// assume
PexAssume.IsNotNull(list);
// arrange
var count = list.Count;
// act
list.Add(item);
// assert
Assert.IsTrue(list.Count == count + 1);
}


3- Parameterized Stubs



If the code-under-test already contains many assertion statements that verify its behavior, an effective parameterized unit test might be quite simple in itself, because it can leverage the assertions in the code.



[PexMethod] 
public void Add( [PexAssumeUnderTest]ArrayList list, object item)
{
list.Add(item);


The attribute PexAssumeUnderTest is a short-hand notation to make sure that the parameter is not null, and has exactly the type indicated by its declaration (not a subtype).





4- Observed Values

At any point in the parameterized unit test, if you want to log the value of any variable or parameter, just use
PexObserve.ValueForViewing("Variable Name", var_name);


5- Allowed Exceptions


Traditional unit test frameworks support the concept of expected exception, where a test case or API call is expected to throw an exception. If the test does not throw the exception or throws an exception that does not match the criteria, the execution fails. The same concept applied to parameterized unit tests.



[PexMethod][PexAllowedException(typeof(ArgumentNullException))] 
void Constructor(string value)
{
// throws ArgumentNullException if value is null
var myClass = new MyClass(value);
}


6- State Relation


This pattern applies when an API call causes an internal state change that can be (partially) observed through other API calls. A classic example of such a pattern is the combination of Insert and Contains operation on any collection type:




[PexMethod]
void InsertContains(stringvalue)
{
var list = new List();
list.Add(value);
Assert.IsTrue(list.Contains(value));
}


7- Roundtrip


This pattern applies to an API that transforms its inputs in a reversible way: When the API has a function f and an inverse function f_1, then it should hold that f_1(f(x))=x for all x. A classic example of such pattern is property setters and getters, where the test fails when the setter rejects a particular argument value.




[PexMethod]
void PropertyRoundtrip(string value)
{
// arrange
var target = new MyClass();

// two-way roundtrip
target.Name = value; // calls setter
var roundtripped = target.Name; // calls getter

// assert
Assert.AreEqual(value, roundtripped);
}


Another example is serialization and deserialization of values.



[PexMethod]
void ToStringParseRoundtrip(int value)
{
// two-way roundtrip
string s = value.ToString();
int parsed = int.Parse(s);

// assert Assert.AreEqual(value, parsed);
}


8- Normalized Roundtrip


The Pattern Roundtrip showed how to test a method for which in inverse operation exists. For example, int.Parse is the inverse of int.ToString. This is not the case in the other direction: int.ToString is not exactly the inverse of int.Parse, because the parsing ignores whitespace (some kind of data normalization):




int.Parse(" 5").ToString() == "5";


This pattern is applied when the API has a function f and an inverse function f_1, then it should hold that f_1(f(f_1(x)))=f_1(x) for all x where f_1(x) is defined.




[PexMethod]
void ThreeWayRoundtrip(string value)
{
// ’hello%20world’ <= ’hello world’
var normalized = Uri.EscapeDataString(value);

// ’helloworld’ <= ’hello%20world’
var intermediate = Uri.UnescapeDataString(normalized);

// ’hello%20world’ <= ’hello world’
var roundtripped = Uri.EscapeDataString(intermediate);

// assert
Assert.AreEqual(normalized, roundtripped);
}



9- Reachability



If your test assumptions are so complicated and it is not clear whether there is any test input that fulfills them. You could use PexAssert.ReachEventually("GoalName") wherever in your test to make sure that there is at least one input that reach this point in your test. The goal name have to be passed to the PexAssertReachEventuallyAttribute constructor. A parameterized unit test fails when Pex does not find a way to reach a goal, indicated by calling the PexAssert.Reached method in a parameterized unit test annotated with the PexAssertReachEventuallyAttribute.




[PexMethod]
[PexAssertReachEventually("passed", StopWhenAllReached = true)]
public void ParsingSuccesful(string input)
{
// complicated parsing code
DateTime date;
if (DateTime.TryParse(input, out date))
{
// and we want to see at least one case where parsing is successful.
PexAssert.ReachEventually("passed");
}
}


Multiple goals can be combined in a single parameterized unit test for more advanced scenarios by passing a list of goal identifiers in the constructor of the PexAssertReachEventuallyAttribute. Each goal identifier must be reached and notified in order for the parameterized unit test to succeed.




[PexMethod]
[PexAssertReachEventually("parsed", "y2k", StopWhenAllReached = true)]
public void ParsingSuccesfulWithMoreGoals(string input)
{
// complicated parsing code
DateTime date;
if (DateTime.TryParse(input, out date))
{
// and we want to see at least one case where parsing is successful.
PexAssert.ReachEventually("parsed");
}
if (date.Year == 2000)
{
// we want to see at least one test with the year 2000
PexAssert.ReachEventually("y2k");
}
}


10- Reachable Implication



Implications assert a property when a predicate (condition) is true. Use PexAssert.ImpliesEventually to make an implication that should be true and that should be executed at least once. The test have to be annotated with the PexAssertReachEventuallyAttribute. A parameterized unit test fails when Pex does not find a way to make the predicate of PexAssert.ImpliesEventually evaluates true at least once.




[PexMethod]
[PexAssertReachEventually]
public void ParseImpliesParse(string input)
{
bool value;
// if TryParse succeeds, Parse should succeed too. Also make sure TryParse succeeds at least once
PexAssert.ImpliesEventually( bool.TryParse(input, out value), () => bool.Parse(input) );
}


11- Seed Values to Help Pex


A parameterized unit test needs concrete input data to be executed. Pex's role is to automatically generate relevant input data via code analysis. Sometimes it might be desirable or even necessary to provide manually chosen seed values to Pex to guide the automated code exploration. In effect, Pex will fuzz the provided values in ways that cause alternative execution paths to be taken. The PexArgumentsAttribute can be used to provide primitive data. Each instance of this attribute gives a list of values that must match the parameter types of the parameterized unit test (and in the same parameters order).




[PexMethod]
[PexArguments("var i = 0;", 0)]
[PexArguments("class Foo {}", 12)]
public void ParseTest(string text, int line)
{
var parser = new Parser();
parser.SetLine(line);
var node = parser.Parse(text);

}

Before analyzing the branch conditions in the code, Pex will first execute the parameterized unit test with the provided values. This way, Pex acquires knowledge about the code reachable from a test, and during the subsequence code exploration Pex will try to further increase code coverage by slightly modifying the values to trigger different execution paths. In effect, Pex will fuzz the provided values.



12- Regression Tests



In these tests some we could persist a computed value in the generated test so when the generated test is executed in the future, it verifies that the (possible changed) code-under-test still computes the same value. There are several ways how outputs can be logged. For a single output value, one can use the return value of the parameterized unit test:



[PexMethod]
public int Add(int a, int b)
{
return a + b;
}

Pex will recursively traverse the observable properties and fields of the value and add assertions in the generated test for each one of them:



[Test]
[PexGeneratedBy(typeof(Program))]
public void Add866()
{
int i;
i = this.Add(0, 0);
PexAssert.AreEqual(0, i);
}

For multiple values, use out parameters in the parameterized unit tests



[PexMethod]
public void Add(int a, int b, out int result)
{
result = a + b;
}

Which Pex will use to generate the following



[Test]
[PexGeneratedBy(typeof(Program))]
public void Add13()
{
int i = 0;
this.Add(0, 0, out i);
PexAssert.AreEqual(0, i);
}

If the number of values might be dynamic, you could log these values using PexObserve.Value and observe it later using the PexObserve.ValueAtEndOfTest method:



[PexMethod]
void Add(int a, int b)
{
int result = a * b;
PexObserve.Value("result", result);
}

Wednesday, April 24, 2013

Getting started with Microsoft Pex

Microsoft Pex is a white box test generation for .NET that came out of Microsoft Research and have been successfully integrated into Visual Studio 2010. It have been a result of collaborative work between Microsoft Research and the Automated Software Engineering Research Group at North Carolina State University led by  Dr. Tao Xie.

You can download and install Microsoft Pex for Visual Studio 2010 from here. We have talked in a previous post about parameterized unit tests and the possibilities it brings. In this post and the following we will explore Microsoft Pex and how it can help you in understanding the input/output behavior of your code, finding inputs that cause the code-under-test to crash, and exploring parameterized unit tests to check whether your code implements the desired functionality for all inputs.

Running Pex for the First Time

In Visual Studio click File > New > Project. In the left pane of the New Project dialog box, click Visual C#. In the center pane, click Console Application, pick a Name for it and click OK. Replace your class Program with the following class. It just the classic triangle classification program :

class Program
{
public static String[] triTypes = { "", // Ignore 0.
"scalene", "isoscelese", "equilateral", "not a valid triangle"};
public static String instructions = "This is the ancient TriType program.\nEnter three integers that represent the lengths of the sides of a triangle.\nThe triangle will be categorized as either scalene, isosceles, equilateral\n or invalid.\n";

public static void Main()
{
int A, B, C;
int T;

Console.WriteLine(instructions);
Console.WriteLine("Enter side 1: ");
A = getN();
Console.WriteLine("Enter side 2: ");
B = getN();
Console.WriteLine("Enter side 3: ");
C = getN();
T = Triang(A, B, C);

Console.WriteLine("Result is: " + triTypes[T]);
Console.ReadLine();
}

public static int Triang(int Side1, int Side2, int Side3)
{
int triOut;

// triOut is output from the routine:
// Triang = 1 if triangle is scalene
// Triang = 2 if triangle is isosceles
// Triang = 3 if triangle is equilateral
// Triang = 4 if not a triangle

// A quick confirmation that it's a valid triangle
if (Side1 <= 0 || Side2 <= 0 || Side3 <= 0)
{
triOut = 4;
return (triOut);
}

// Detect any sides of equal sides
triOut = 0;
if (Side1 == Side2)
triOut = triOut + 1;
if (Side1 == Side3)
triOut = triOut + 2;
if (Side2 == Side3)
triOut = triOut + 3;
if (triOut == 0)
{
if ((Side1 + Side2 <= Side3) || (Side2 + Side3 <= Side1) || (Side1 + Side3 <= Side2)) // confirm that it is a valid triangle
triOut = 4;
else
triOut = 1;
return (triOut);
}

if (triOut > 3)
triOut = 3;
else if ((triOut == 1) && (Side1 + Side2 > Side3))
triOut = 2;
else if ((triOut == 2) && (Side1 + Side3 > Side2))
triOut = 2;
else if ((triOut == 3) && (Side2 + Side3 > Side1))
triOut = 2;
else
triOut = 4;
return (triOut);
}

In the Build menu, click Build Solution.


To run Pex on your code, right-click in the body of the Triang method, and click Run Pex. If this is your first time running Pex, the Pex: Select a Test Framework dialog box appears. You could select your preferred test frame (Visual Studio Unit test, or NUnit)  and provide the installation path for its, then click OK. This dialog box will not appear again after you select the test framework. After a brief pause, Pex shows the results of its analysis in the Pex Exploration Results window. When you run Microsoft Pex on your .NET code, Pex generates test cases by analyzing the code-under-test. For every statement in the code, Pex will eventually try to create a test input that will reach that statement. Pex will do a case analysis for every conditional branch in the code—for example, if statements, assertions, and all operations that can throw exceptions. Each row in the table contains input/output values for the method under consideration(Traing). In the Pex Exploration Results window, click one row in the table on the left to see details in the right pane. You could select these rows and save them as unit tests. These details also displayed on the right in the Pex Explorer pane as test cases. ConsoleApplication2 - Microsoft Visual Studio_2013-04-23_16-45-40


On the next post we will explain why Pex chose these values, and other Pex stuff :)

Monday, April 22, 2013

Unit Tests vs Parametrized Unit Tests

Unit Tests

Using the conventions of NUnit unit tests as test methods contained in test classes. A parameterless method decorated with a custom attribute like [TestMethod] is a test method. Usually, each unit test explores a particular aspect of the behavior of the class-under-test.

Here is a unit test written in C# that adds an element to a .NET ArrayList instance. The test first creates a new array list, where the parameter to the constructor is the initial capacity, then adds a new object to the array list, and finally checks that the addition was correctly performed by verifying that a subsequent index lookup operation returns the new object.

[TestMethod] 
void TestAdd() 
{ 
 ArrayList a = new ArrayList(0); 
 object o = new object(); 
 a.Add(o); 
 Assert.IsTrue(a[0] == o); 
}

It is important to note that unit tests include a test oracle that compares observed behavior with expected results. By convention, the test oracle of a unit test is encoded using assertions. The test fails if any assertion fails or an exception is thrown. Unit test frame- works can also deal with expected exceptions. For a quick introduction about NUnit refer previous posts.

A test suite produced by unit testing with high code coverage gives confidence in the correctness of the tested code. However, writing unit tests to achieve high coverage can be time-consuming and tedious given that test execution frameworks only automate the test executions. You still have to write your test cases !

To address this problem, several automatic unit test generation tools such as Parasoft JTest or jCUTE can automatically generate conventional unit tests. These tools, nevertheless, cannot guarantee high code coverage, unless testers manually write some tests.

 


Parametrized Unit Tests


The unit test above specifies the behavior of the array list by example. Strictly speaking, this unit test only says that by adding a new object to an empty array list, this object becomes the first element of the list. What about other array lists and other objects?

[TestAxiom]
void TestAdd(ArrayList a, object o) 
{
 Assume.IsTrue(a!=null);
 int i = a.Count;
 a.Add(o);
 Assert.IsTrue(a[i] == o);
}

By adding parameters we can turn a closed unit test into a universally quantified conditional axiom that must hold for all inputs under specified assumptions.

Adding parameters to a unit test improves its expressiveness as a specification of intended behavior, but we lose concrete test cases. We can no longer execute this test axiom by itself. We need actual parameters. But which values must be provided to ensure sufficient and comprehensive testing? Which values can be chosen at all?

In the ArrayList example, if we study the internal structure of the .NET Framework implementation, we observe that there are two cases of interest. One occurs when adding an element to an array list that already has enough room for the new element (i.e. the array list’s capacity is greater than the current number of elements in the array list). The other occurs when the internal capacity of the array list must be increased before adding the element.

If we assume that library methods invoked by the ArrayList implementation are themselves correctly implemented, we can deduce that running exactly two test cases is sufficient to guarantee that the parametrized TestAdd(...) succeeds for all array lists and all objects.

[TestMethod]
void TestAddNoOverflow() 
{
 TestAdd(new ArrayList(1), new object());
}

[TestMethod]
void TestAddWithOverflow() 
{
 TestAdd(new ArrayList(0), new object());
}

Splitting axioms and test cases in this way is a separation of concerns:


  • First, we describe expected behavior as parametrized unit tests.
  • Then we study the case distinctions made by the code paths of the program under test to determine which inputs make sense for testing.

So, parametrized unit tests are more general specifications than conventional unit tests because it specifies the behavior for the whole input classes other than for a single concrete value of the input. But it needs concrete parameter values to be executed.


The good news is, Given a parametrized unit test, a test-generation tool, such as Microsoft Pex, can automatically generate tests with concrete inputs for the parameters to achieve high coverage. Pex explores the behaviors of a parametrized unit test using a technique called dynamic symbolic execution. Dynamic Symbolic Execution (DSE) is a variation of symbolic execution, which systematically explores feasible paths of the program under test by running the program with different test inputs to achieve high structural coverage. It collects the symbolic constraints on inputs obtained from predicates (condition statements like if else, switch,…) in branch statements along the execution and relies on a constraint solver [like Zap or Simplify] to solve the constraints and generate new test input for exploring new path. For each set of concrete test input that leads to a new path that achieves new coverage, Pex generates a corresponding conventional unit test. So, all what you have to do is write your parametrized unit test.


In this post we gave a brief overview about parametrized unit tests and its related research areas. It’s a very important concept to understand before start learning Microsoft Pex, which we will start . In later posts, we will explore Microsoft Pex in detail.