Visual Studio Team System Unit Testing

Of all the features in Visual Studio Team System (VSTS), none is more controversial than the unit testing features. The community campaign to convince Microsoft to move unit testing from the premium-level VSTS product into a more readily accessible version has been unsuccessful so far, and many are legitimately concerned that the requirement for VSTS will significantly decrease the uptake of the unit testing features. For all the controversy, the actual functionality of the unit testing has not received a lot of attention. This article examines the VSTS Unit Test offerings from a Visual C++ perspective.

C++ has had the CppUnit unit testing framework available for many years, but it seems like unit testing has not made the inroads in the C++ community that it has in .NET and particularly Java. The reasons for the slow uptake have been varied—native C++ lacks the metadata that makes hooking up unit tests easy and intuitive, CppUnit has not been as polished as its managed contemporaries, and the agile mindset that has been the driving force behind unit testing has not been as globally applicable in the C++ world where large, established projects are more prevalent. VSTS unit testing cannot solve the two intrinsic problems with C++ unit testing (cultural issues related to agile programming and lack of metadata in native C++), but it does deliver a nicely polished unit testing framework that is ready to roll with C++.

Thankfully, the few caveats that do exist with VSTS Unit Testing and C++ are fairly obvious:

  • To conduct unit tests for native code, a standard native entry point into that code needs to be exposed.
  • The unit testing execution engine in VSTS is designed around managed code and managed code attributes. Hence, native code cannot be used to write unit tests.
  • Code generation of unit tests is supported only if the production code (i.e., the code that is to be tested) is compiled with the /clr:safe option.

The fact that the set of limitations is so small is quite impressive—the VSTS team has pushed the functionality right up to the limits of what the language and the .NET and Windows platforms support. The full breakdown of the entire unit testing and C++ combinations and their behaviors is available on the MSDN site.

Authoring and Executing Unit Tests

For those familiar with unit testing frameworks like NUnit, the VSTS unit testing functionality will seem quite familiar. A unit test is a managed method that is marked with the TestMethod attribute. Unit tests must be included in a type with the TestClass attribute. Unit tests execute some type of logic and, at the end of executing this logic, compare the result achieved with an expected value using conditional logic in the Microsoft.VisualStudio.TestTools.UnitTesting,Assert class. If the comparison executed through the Assert class is correct, the unit test passes. Otherwise, it fails.

There is obviously a lot of boilerplate code required before the real functionality of the unit tests can be added. That is where the code generation abilities of VSTS come in. The simplest way to add a unit test for a method is to bring up the context menu in the text editor by right-clicking near the method, and selecting Create Unit Tests. (see Figure 1).

Figure 1. Adding a Unit Test to a C++ Project

Note that this feature will be available only if the code that is being unit tested (referred to as production code in the MSDN documentation) is compiled with /clr:safe. For the simple method shown in Figure 1, the unit test generated will look as follows:

[ TestMethod ]
void AddTest()
{
   int i;    // TODO: Initialize to an appropriate value

   int j;    // TODO: Initialize to an appropriate value

   int expected;
   int actual;

   actual = CLRLibrary::Class1::Add(i, j);

   Assert::AreEqual(expected, actual,
                    L"CLRLibrary.Class1.Add did not return the
                    expected value.");
   Assert::Inconclusive(L"Verify the correctness of this test method.");

}

Notice that the generated test calls Assert::Inconclusive to let the unit-testing framework know that the test shouldn't be counted as a passed test. The default result for a test in the absence of a call to an Assert class method is a pass, which can be useful if a test is written purely to check that the tested code runs without throwing an exception (if an unhandled exception is thrown, the test will be defined as a failure). Modifying the unit test so that it actually tests the Add method correctly would result in the following code:

[ TestMethod ]
void
AddTest()
{
   int i = 5;
   int j = 6;

   int expected = 11;
   int actual = CLRLibrary::Class1::Add(i, j);

   Assert::AreEqual(expected, actual, L"CLRLibrary.Class1.Add did" +
                    " not return the expected value.");
}

Unit tests can be run directly from within Visual Studio by selecting either Start Selected Test Project With Debugger or Start Selected Test Project Without Debugger.

Managing Tests

Adding a test project to a solution will also result in the creation of a configuration file that defines the various configuration properties for running tests. Thankfully, VSTS also ships with a Visual Studio Addin for managing the configuration file. Double-clicking on the file in Visual Studio will bring up the dialog shown in Figure 2.

Figure 2. Configuring Unit Tests

This dialog allows tests to be configured to run on remote machines; it configures code coverage settings; and it allows scripts for setting up and tearing down before and after a test run to be executed.

VSTS has a great deal of flexibility in selecting which tests to run. The simplest way to configure which tests to run is through the Test Manager. Selecting Tests | Windows | Test Manager will bring up the Test Manager, as shown in Figure 3.

Figure 3. Test Manager

It is possible to create a filter that runs tests only with a field that matches a particular keyword. You also can create a custom Test List and drag-and-drop tests into the List. Once a filter or Test List has narrowed the tests that need to be run, the selected tests can be run with or without the debugger. The test management settings built using the Test Manager are stored in a Visual Studio Metadata file, which is an XML file with a vsmdi extension. This file is stored with the solution, and settings made in the Test Manager are not lost when it is closed.

The test configuration file dialog shown in Figure 2 can be used to instrument binary files for code coverage analysis. Once the instrumentation has been added and the tests re-run, the code coverage results can be brought up with a context menu in the Test Results window, as shown in Figure 4.

Figure 4. Accessing Code Coverage Results

Unit Testing C++ Made Easy

For those that have found getting into unit testing with C++ too hard, the release of VSTS represents an excellent opportunity to have another shot at integrating unit testing into your development practices.

About the Author

Nick Wienholt is an independent Windows and .NET consultant based in Sydney, Australia. He is the author of Maximizing .NET Performance from Apress, and specializes in system-level software architecture and development with a particular focus on performance, security, interoperability, and debugging. Nick can be reached at NickW@dotnetperformance.com.



Comments

  • Unit test for C++ win32 application

    Posted by sira on 12/07/2012 12:37am

    First time I learn about unmanaged C++ unit test from this tutorial. Defnitly this will help to beginers http://codeketchup.blogspot.sg/2012/12/unit-test-for-unmanaged-c-in-visual.html

    Reply
  • Using printf in VC++ unit tests

    Posted by Roger Pearse on 10/11/2012 12:48am

    One thing that will make you hate Microsoft; things like printf produce no output. Google searches will bring up stuff like OutputDebugString() which will not work either. What you have to use is Console::WriteLine(L"myoutput string");

    Reply
  • .NET performance

    Posted by Annie Calvert on 08/08/2012 02:00am

    It's not a profiler but an Application Performance Management Solution that gives you the ability to trace every single transaction in your application - not only on one CLR but across all CLRs you have in your system. http://www.dapfor.com/en/net-suite/net-grid/features/performance

    Reply
  • Can I use unit-tests in MFC-projects?

    Posted by michot on 05/16/2006 10:21pm

    Can I use unit-tests in MFC-projects? Does Visual Studio Team System distributed within Visual Studio 2005 Pro Ed? Thank you for your answers.

    • RE: Stepping in to unit tests

      Posted by sheyenne on 12/15/2009 02:04am

      Two potential issues that could cause this: * The symbols are being loaded for some reason (check using the Modules Debug Window) * Native debugging isn't enabled

      Reply
    • Stepping in to unit tests

      Posted by metaforge on 12/14/2009 07:52pm

      I've got a native C++ DLL project and I've written an extern mC++ NUnit project that can call into the native code, get a result, and test its correctness. However, when I run the test in debug mode, it doesn't seem to let me step into the native code - breakpoints in the native code are greyed out saying no symbols loaded. I do have the native code project in the same solution, and if I write a native command line exe project, I can call the DLL and step into it. Any ideas?

      Reply
    • RE: Can I use unit-tests in MFC-projects?

      Posted by sheyenne on 05/17/2006 02:44am

      You can test MFC projects, but you need to expose an entry point via a C-style DLL interface or COM to the methods you want to test. You also need VSTS - VS Pro won't do it.

      Reply
    Reply
Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Managing your company's financials is the backbone of your business and is vital to the long-term health and viability of your company. To continue applying the necessary financial rigor to support rapid growth, the accounting department needs the right tools to most efficiently do their job. Read this white paper to understand the 10 essentials of a complete financial management system and how the right solution can help you keep up with the rapidly changing business world.

  • Event Date: April 15, 2014 The ability to effectively set sales goals, assign quotas and territories, bring new people on board and quickly make adjustments to the sales force is often crucial to success--and to the field experience! But for sales operations leaders, managing the administrative processes, systems, data and various departments to get it all right can often be difficult, inefficient and manually intensive. Register for this webinar and learn how you can: Align sales goals, quotas and …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds