Automated White Box Testing with Microsoft PEX Framework

Introduction

Microsoft’s PEX and Moles are two frameworks that can generate test suites with high code coverage. The PEX Framework from Microsoft is an automated White Box Testing tool that can be integrated with Visual Studio IDE seamlessly — it can simply work as a Visual Studio add-in. This article presents how this tool can be used in your applications and discusses why it is useful.

Pre-Requisites

To work with Microsoft PEX framework, you should have the one of the following installed in your system:

  • Microsoft Visual Studio 2005
  • Microsoft Visual Studio 2008 with SP1
  • Microsoft Visual Studio 2010

Note that if you are using Visual Studio Express Edition, you wouldn’t be able to use PEX as an add-in. In this case, you would need to use PEX from the command line.

You can download a copy of Microsoft PEX framework from this link: http://visualstudiogallery.msdn.microsoft.com/en-us/905980ee-039a-4192-bd6c-81e6ec2978c7

Black Box and White Box Testing — A Quick Look

White Box and Black Box are two of the widely used types of testing software applications. While Black Box testing involves an external view of the test object, White Box testing is a type of testing that is done by the developer of the code. It requires knowledge of how the code works and how it is structured. In Black Box testing you have lesser code coverage and doesn’t consider all scenarios. In essence, Black Box testing involves testing the functionality of the application as opposed to testing how the internal program structures work.

White Box testing has more code coverage and helps you find more potential errors in your application code. If used properly, White Box testing can help you get a more stable build for your application. The Wikipedia states: “White-box testing (a.k.a. clear box testing, glass box testing, transparent box testing, or structural testing) is a method of testing software that tests internal structures or workings of an application, as opposed to its functionality (i.e. black-box testing). In white-box testing an internal perspective of the system, as well as programming skills, are required and used to design test cases. The tester chooses inputs to exercise paths through the code and determine the appropriate outputs.” Reference: http://en.wikipedia.org/wiki/White-box_testing

Understanding PEX — where does PEX fit in?

Microsoft PEX framework is a Visual Studio add-in that can be used for testing applications that are targeted at the managed environment of the CLR. Microsoft’s PEX framework can help improve the quality of the software with minimal developer effort. It can automatically generate almost all possible code paths in your application’s code. Moreover, PEX provides support for parameterized unit testing – an extension of traditional unit testing that helps you reduce the test maintenance costs in your project.

The PEX Framework is a Visual Studio add-in that can provide runtime code analysis with minimal developer effort. With this framework, you can easily understand the input output behaviour in the application’s source code, create and save automated tests and even write parameterized unit tests and generate test suites seamlessly.

If PEX is installed in your system, you can run PEX by right-clicking on the method on which you would want PEX to be executed and then selecting “Run Pex” from the context menu. You can execute PEX on any method in your application. Note that the PEX attributes and helpers are all contained inside the Microsoft.Pex.Framework.dll assembly. PEX generally generates a test suite that allows for maximum code coverage.

When working in Unit Testing mode, PEX explores the code for any method that contains the PexMethodAttribute attribute as shown below:

  using Microsoft.Pex.Framework;
  [PexMethod]

  public static void Extract(string str, int n)
  {
     String result = str.Substring(0,n);
  }

But before we delve deep into the discussions, let’s take a quick tour of what unit testing is all about. So, what is a unit test anyway? A unit test may be defined as a program that can check an aspect of the implementation that is to be tested. A unit is the smallest testable portion of a particular program that is under test. A parameterized unit test is essentially a method that accepts parameters, executes the code to be tested and states assertions. The MSDN states: “The primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect. Each unit is tested separately before integrating them into modules to test the interfaces between modules. Unit testing has proven its value in that a large percentage of defects are identified during its use.” Reference: http://msdn.microsoft.com/en-us/library/aa292197(VS.71).aspx

To generate parameterized unit test stubs, you can simply right click on the class or the method on which you intend to generate the stubs in your Visual Studio IDE. Such parameterized methods are generally placed inside a type that acts as a test class for unit testing. Here is an example:

  using Microsoft.VisualStudio.QualityTools.UnitTestFramework;

  [TestClass]

  public partial class Utilities
  {

   [PexMethod]

   public static void Extract(string str, int n)
   {
      String result = str.Substring(0,n);
   }

  }

The test class may be in turn be marked with the PexClassAttribute to imply that this is the type under test that contains parameterized unit testing methods. The test class shown earlier may be decorated with this attribute as shown in the code listing that follows:

  using Microsoft.VisualStudio.QualityTools.UnitTestFramework;

  [TestClass]

  [PexClass(typeof(Utilities))]

  public partial class Utilities
  {

   [PexMethod]

   public static void Extract(string str, int n)
   {
      String result = str.Substring(0,n);
   }

  }

Summary

PEX, an automated white box testing framework from Microsoft research, is a great tool that helps in understanding the behaviour of your application’s code, helps in debugging and can create a test suite for you that covers all corner cases automatically. This article provided a head start to understanding and working with Microsoft PEX framework for building applications that are error free and robust. Happy reading!

Suggested Readings

http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/Reference%20Manual.html
http://channel9.msdn.com/tags/PEX/
http://msdn.microsoft.com/en-us/magazine/ee819140.aspx
http://research.microsoft.com/en-us/projects/pex/gettingstarted.aspx
http://research.microsoft.com/en-us/projects/pex/
http://research.microsoft.com/en-us/projects/pex/downloads.aspx
http://www.softwaretestinghelp.com/white-box-testing/

Related Articles

More by Author

Must Read