Data Driven Testing with Visual Studio Team Suite

Microsoft Visual Studio Team Suite (VSTS) seamlessly integrates the development environment with a suite of robust testing products that make developers more efficient when testing their applications. Ease of use and integration with the IDE are two critical factors to look for when examining a software testing product and VSTS offers both. VSTS provides the capability to generate skeleton code for unit tests with just a couple of mouse clicks, and without having to ever leave the IDE. This means that developers can create more effective unit tests faster, and because they never even have to leave the IDE, developers create their tests in an environment they are used to, without requiring a third-party tool. Because of this familiarity, developers will be more likely to use the tools to create rigorous tests to ensure the quality of their software.

One of the new capabilities provided by VSTS is data-driven testing. With data-driven tests, developers can connect their unit and Web tests to data sources that can be used to drive their test cases. This article will discuss data-driven testing for both unit tests and Web tests and how your organization can realize the benefits of these capabilities.

Data-Driven Unit Tests

One common test scenario involves the tricky business rules surrounding the validity of data input provided by a user. For example, a business rule such as A customer cannot have an address that is a P.O box unless they are outside the U.S. might be a requirement given to the development team by business users. Normally, a developer would have to spend time coding a unit test that validates this specific business rule. However, with data-driven unit testing, you can model the input to your application using an outside data source and then can use this data to drive your testing.

The sample application that has been created for this article is a contact manager Web application that provides some basic functionality such as adding a new contact, viewing, a list of current contacts and deleting a contact. You will learn how to use data-driven testing to validate the business rules of this application by using both unit tests and Web tests by simulating user input into the system.

Generating Unit Tests with VSTS

For this example, I want to create unit tests for my ContactInfoManager class that provides the functionality for adding, removing, and retrieving ContactInfo objects. The four methods I want to generate unit tests for are: Add(), DeleteByID(), GetNumContacts(), and GetAllContacts().

To generate the Unit Test skeleton code, right-click in my code file and select "Create Unit Tests." as shown in Figure 1.

Figure 1: Create Unit Tests

Then, select the methods to generate unit tests for and the project to output the tests to (See Figure 2).

Figure 2: Choosing Methods to Generate Test Cases

After clicking OK, four test methods are added to the ContactInfoManagerTest project in the ContactInfoManagerTest.cs file. (If the project is not already created, VSTS will prompt you to create it.) By default, each of them has an Assert.Inconclusive() method as their last statement. If your team is using Test Driven Development, you might find that this behavior is incongruous with the TDD mantra of Red-Green-Refactor because an inconclusive assertion always displays a yellow icon instead of a red icon that TDD practitioners normally expect. Thankfully, this behavior can be disabled by clicking the "Settings" button, and un-checking the "Mark all tests inconclusive by default" checkbox.

Figure 3: Generating Unit Test Code

After clicking OK, VSTS generates the skeleton code for these unit tests and adds them to the test project. Execute the tests from within the Test Manager by clicking the "Play" button.

Figure 4: Test Manager Button

Because the code for the methods under test has not yet been implemented, all the tests will fail.

Figure 5: Failed Tests

VSTS will generate skeleton code for the unit tests; however, it is still up to the developer to implement the unit tests effectively. For information on how to write a meaningful unit tests, please see this MSDN article.

The Application under Test

The application under test is a Contact Manager application. The specific functionality under test exists in two classes, ContactInfo and ContactInfoManager. The ContactInfo class encapsulates all of the data for a single contact (Name, address, and so forth). The ContactInfoManager class encapsulates the business logic surrounding the contact manager application. The class diagrams for ContactInfo and ContactInfoManager are below.

   

Figure 6: Class Diagrams of ContactInfo and ContactInfoManager

I want to use data driven tests to verify that only invalid contact info records will be rejected by the application. I also want to ensure that valid contact info records will always be accepted and added. In this application, the validity of the ContactInfo object is determined by several simple rules:

  1. The ContactInfo object is not null.
  2. The name property of the ContactInfo object is not null and has a length > 0.
  3. If a phone number is present, it must be exactly 10 numeric characters.
  4. If a zip code is present, it must be exactly 5 numeric characters.
  5. If a state is present, it must be the 2 character abbreviation.

Now, I will create two test methods to verify these rules, one called TestValidContactInfo(), which verifies that valid contact info records are always added to the system, and another called TestInvalidContactInfo(), which verifies that invalid records are rejected by the system. These methods build a ContactInfo object from the data provided in the aforementioned tables, and then pass this object to the ContactInfoManager's Add() method. The Add() method handles the validation of the ContactInfo object and is implemented to return the ID of the ContactInfo object if it is successfully added, or -1 if the record is rejected.

Connecting the Unit Tests to a Data Source

Before going any further implementing the unit tests, the tables that will be used to drive my tests need to be created. In a SQL Server 2005 database, I have created two tables (ValidContactInfo to be used by the TestValidContactInfo() unit test and InvalidContactInfo to be used by the TestInvalidContactInfo() unit test). As you can see in Figures 7 & 8, each table's columns correspond directly to one of the ContactInfo object's properties. The ValidContactInfo table is populated with data values that are valid and should always be accepted and added to the application.

Figure 7: Valid Data

The InvalidContactInfo table is populated with values that are invalid and should be rejected by the application.

Figure 8: Invalid Data

Note: In this case, I have used a SQL Server 2005 Database as my data source. However, you can use a variety of different data sources such as Access files, any ODBC data source, or an Oracle Database.

Now that the tables have been created and populated, I want to connect their corresponding unit tests to them. Unit tests can be connected to a data source from the Test Manager view by clicking on the test to be bound to a data source and then modifying the "Data Connection String" and "Data Table Name" values in the Unit Test Properties window to point to the desired data source and table.

Figure 9: Unit Test Properties

This adds a DataSource attribute to the unit test method. Also, you can decorate the test methods with the DataSource attribute yourself, if you so choose.

Figure 10: Data Source Attribute

Now that the unit tests are connected to their data sources, it is time to write the code to implement the tests.

The TestValidContactInfo unit test uses data from its table to build ContactInfo objects that are passed to the Add() method of the ContactInfoManager class. This test verifies that valid contact information is always successfully added to the application. Unit tests access the data in their data source through the TestContext.DataRow property that stores the value of the current row in the data source as you iterate through all of the rows and is of type System.Data.DataRow. The source listing below illustrates how the ContactInfo object is populated with values from the current row in the data source.

[DataSource("System.Data.SqlClient",
            "Data Source=VSTS;Initial Catalog=ContactManagerWebTest;
             Integrated Security=True", "ValidContactInfo",
            DataAccessMethod.Sequential), TestMethod()]
public void TestValidContactInfo()
{
   ContactInfoManager manager = ContactInfoManager.Instance;
   ContactInfo ci = new ContactInfo();
   ci.Name = TestContext.DataRow["Name"].ToString().Trim();
   ci.AddressLine1 =
      TestContext.DataRow["AddressLine1"].ToString().Trim();
   ci.AddressLine2 =
      TestContext.DataRow["AddressLine2"].ToString().Trim();
   ci.City     = TestContext.DataRow["City"].ToString().Trim();
   ci.Email    = TestContext.DataRow["Email"].ToString().Trim();
   ci.Phone    = TestContext.DataRow["Phone"].ToString().Trim();
   ci.State    = TestContext.DataRow["State"].ToString().Trim();
   ci.ZipCode  = TestContext.DataRow["ZipCode"].ToString().Trim();
   ci.HomePage = TestContext.DataRow["HomePage"].ToString().Trim();

   //manager will return a new id for this Contact
   int contactId = manager.Add(ci);

   //assert that it is not -1
   Assert.IsFalse(contactId == -1);
}

The TestInvalidContactInfo unit test builds ContactInfo objects and passes them to the Add() method of the ContactInfoManager class. This test will be used to verify that no invalid contact information can be added to your system.

[DataSource("System.Data.SqlClient",
            "Data Source=VSTS;Initial Catalog=ContactManagerWebTest;
             Integrated Security=True",
            "InvalidContactInfo", DataAccessMethod.Sequential),
            TestMethod()]
public void TestInvalidContactInfo()
{
   ContactInfoManager manager = ContactInfoManager.Instance;
   ContactInfo ci = new ContactInfo();
   ci.Name = TestContext.DataRow["Name"].ToString().Trim();
   ci.AddressLine1 =
      TestContext.DataRow["AddressLine1"].ToString().Trim();
   ci.AddressLine2 =
      TestContext.DataRow["AddressLine2"].ToString().Trim();
   ci.City     = TestContext.DataRow["City"].ToString().Trim();
   ci.Email    = TestContext.DataRow["Email"].ToString().Trim();
   ci.Phone    = TestContext.DataRow["Phone"].ToString().Trim();
   ci.State    = TestContext.DataRow["State"].ToString().Trim();
   ci.ZipCode  = TestContext.DataRow["ZipCode"].ToString().Trim();
   ci.HomePage = TestContext.DataRow["HomePage"].ToString().Trim();

   int contactId = manager.Add(ci);
   int invalidContactId = -1;
   Assert.IsTrue(contactId == invalidContactId);
}

The tests are based off of the requirement that the Add() method returns a unique ID of the newly inserted ContactInfo. If the ContactInfo is invalid, and thus not added, the method returns -1.

Data Driven Testing with Visual Studio Team Suite

After writing the tests, I implemented the Add() method (see code below) to fulfill the requirements of the tests and then ran them again.

public int Add(ContactInfo ci)
{
   ContactInfoValidator civ = new ContactInfoValidator(ci);
   int contactId = -1;
   if (civ.Validate())
   {
      contactId = this.DataManager.Insert(ci);
   }
   return contactId;
}

When the data-driven unit test is executed, it will run once for each row of its data source. Data-driven tests can be set to execute either sequentially or randomly by changing the value of the "Data Access Method" unit test property in the Test Manager view, or by modifying the DataSource attribute (see Figure 10). If the test fails for any of the rows in the data source, the entire unit test will show that it failed. Conversely, the unit test will only pass if it passed for all of the rows in its data source.

[Driven21.jpg]

Figure 11: Unit Tests Passed

You can view the details of a test by double-clicking on the result in the Test Results pane. This shows the test results (pass/fail) for each row in the data source.

[Driven23.jpg]

Figure 12: Data-Driven Unit Test Results

Data-Driven Web Testing

In addition to creating data-driven unit tests to test the business logic of the application, I would like to go one step further and perform data-driven Web testing to test the Web interface. I have created a simple Web form that has one text box for each property of the ContactInfo object.

Figure 13: Contact Manager Web Form

When the user adds a new contact, the ContactInfo object is constructed and its properties are populated and passed to the ContactnfoManager's Add() method that was tested above. If the contact is successfully added, a label on the Web form displays a success message. If the contact is rejected, the label displays a failure message. I could test this functionality by manually entering data and clicking save myself, but that seems like it would be wasted effort. Instead, I am going to use the same technique displayed above to automate testing with a data-driven Web test.

The steps you will follow for the Data-Driven Web Testing process are as follows:

  1. Create the Web Test.
  2. Record a browser session.
  3. Add a data source to the Web test.
  4. Bind form post parameters to the data source.
  5. Add validation rule to verify if this test passes or fails.
  6. Set the run settings to execute once for each row in my data source.
  7. Run the Web test.

Creating a Data-Driven Web Test

Create a new Web test by clicking "New Test" under the "Test" menu item.

Figure 14: Creating a New Test

From there, you will be prompted to select the type of test to create. Select "Web Test" and click OK.

Figure 15: Creating a Web test

This loads a browser window with an additional pane called the "Web Test Recorder" that is used to record Web testing sessions.

Figure 16: Web Test Recorder

Because all I am testing is the "Add" functionality of the Web form, I populate the text boxes with test data and click the "Add Contact" button. This is all that needs to be recorded for this test. After clicking the "Stop" button on the "Web Test Recorder," the details of the Web test are displayed.

As you can see from the Web test details, any values entered in the text boxes during the recording are stored under the "Form Post Parameters" of the Web test. If I ran the Web test again, it would execute the recorded Web session, submitting all the values that are stored in the form post parameters to the corresponding form fields.

Figure 17: Form Post Parameters

I want this test to use the data from the InvalidContactInfo table used in the unit test example to verify that the Web form rejects all invalid contact info. To do this, click the "Add Data Source" icon, enter your connection information, and click OK. Then, you will be prompted to select the tables to add as the data source. Select the InvalidContactInfo table.

Figure 18: Add Data Source Button

Now that you have added the data source to your Web test, the next step is to bind the form post parameters to their corresponding values in the data source. This is accomplished by modifying the properties of the form parameters. For each of the form post parameters, change the "Value" setting to the corresponding column of the data source.

Figure 19: Binding Form Post Parameters to Data Source

For instance, the tbName form post parameter should be bound to InvalidContactInfo.Name, tbAddressLine1 should be bound to InvalidContactInfo.AddressLine1, and so on. When completed, all of the form post parameters should be bound to their corresponding column in the data source.

Figure 20: All Form Post Parameters have been bound

The final step in creating this data-driven Web test is to define the success criteria for the test by creating a simple Validation Rule that looks for the text "Cannot add invalid contact info into the system" after the save button is clicked. Validation rules can be used to check form fields, text on a page, request times, required attribute values and required tags, and are used to verify whether or not a Web test has passed or failed. More information on Validation Rules is available here.

Now that the Web test data bound and ready to go, it's time to execute it. Select the "Run Test (Pause Before Starting)" option from the Play button.

Figure 21: Running the Data-driven Web test

This is done because you want to set the test run to execute once for each row in the data source before it executes. Click the "Edit run settings" link and then check the "One run per data source row" radio button in the Web Test Run settings.

Then, click the Play button to execute the data-driven Web test. The test will run once for each row in the data source, and will use the validation rule to check whether or not that run has passed or failed. Detailed run results are available under the "Details" tab.

Figure 22: Web Test Results

To increase or decrease the number of runs that are executed, simply add new rows to the data source. No new code modifications are required.

Conclusion

The integrated testing tools provided by Microsoft Visual Studio Team Suite can be used to drastically improve the overall quality of software developed by your organization. This article has examined data-driven unit and Web tests and how they can be used to create more effective tests and improve developer efficiency during the testing process.

About the Author

Dan Gartner is a Senior Consultant with Crowe Chizek and Company, LLC and lives in Chicago, IL. He specializes in designing and developing applications for rapidly evolving organizations using Microsoft technologies.



Comments

  • There are no comments yet. Be the first to comment!

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • The explosion in mobile devices and applications has generated a great deal of interest in APIs. Today's businesses are under increased pressure to make it easy to build apps, supply tools to help developers work more quickly, and deploy operational analytics so they can track users, developers, application performance, and more. Apigee Edge provides comprehensive API delivery tools and both operational and business-level analytics in an integrated platform. It is available as on-premise software or through …

  • Not long ago, security was viewed as one of the biggest obstacles to widespread adoption of cloud-based deployments for enterprise software solutions. However, the combination of advancing technology and an increasing variety of threats that companies must guard against is rapidly turning the tide. Cloud vendors typically offer a much higher level of data center and virtual system security than most organizations can or will build out on their own. Read this white paper to learn the five ways that cloud …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds