Application Security Testing: An Integral Part of DevOps
Introduction to Team Foundation Server
Modern software development uses automated testing to make sure the quality of the final product is high with minimum manual effort. The easiest solution to testing automation is unit testing, where numerous small tests are executed in an automated fashion. When the unit tests are run often enough and they test the application code thoroughly, these tests can become a quality guarantee. The guarantee is that application still works after code modifications.
Microsoft Visual Studio itself contains nice support for different types of tests, especially in the latest 2010 version. Although you can get far with Microsoft Visual Studio alone, the combination of Visual Studio and Team Foundation Server 2010 seems to be the best technology solution Microsoft can give.
Understanding Your Testing Options
Since most developers and testers are already familiar with Microsoft Visual Studio, the tool itself is a good starting point to explore testing options. If you go to the New Project dialog box in Visual Studio and navigate to the Test template group on the left (see Figure 1), you can see a single icon titled Test Project.
Figure 1. Creating a test project starts through the New Project dialog box.
This icon indicates that you will have to start a special testing project in Microsoft Visual Studio to get access to the different test types Visual Studio supports. Oftentimes, this kind of testing project is simply one of the projects you have in a Visual Studio solution (the .sln file). For instance, if you are writing ASP.NET MVC applications, you usually have a project in your solution that is already a test project and linked to the actual MVC application (Figure 2).
Figure 2. A test project is an integral part for ASP.NET MVC applications, for instance.
Whenever you have a testing project in place, Microsoft Visual Studio allows you to create a wide range of tests, depending on your Visual Studio edition. Visual Studio 2010 comes in three commercial editions: Professional, Premium and Ultimate. The higher you climb in the ladder, the more testing options you have. To add a new test to your test project, simply use the Test/New Test menu command (alternatively, you can use the commands in the project menu or through the "Add New Item" dialog box). Then, simply select the test you want (Figure 3).
Figure 3. Visual Studio 2010 Ultimate contains many different testing options.
In the Ultimate edition of Microsoft Visual Studio, you have a broad selection of testing options from traditional unit tests to testing your database, web applications, user interface, and even load testing.
For instance, you might have an ASP.NET web application which you want to test. You might then create a coded UI test, which allows you to record the steps when you work with your web browser. Then, you would simply click through the application to record a testing script (Figure 4). Once done, Microsoft Visual Studio will create testing code for you, and your task would be to set the conditions through which you know that you application worked correctly. For instance, a condition might check whether a certain string can be found from the HTML code returned by the server application.
Figure 4. Web tests can be created by recording a coded UI test script.
These possibilities are actually part of Visual Studio itself, and do not require Team Foundation Server. What is the role of Team Foundation Server in testing, then? The real power of TFS starts to show when you want to automate your testing. Let's take a simple example of testing evolution. First, a developer company decides to start using unit tests. In the beginning, the tests are run by hand by the developers themselves. Soon, the number of tests grows, and running them all becomes more and more time-consuming. In the end, the company chooses to automate the testing and run the tests nightly. This way, the developers have a fresh set of testing reports each morning when they come to the office. Automation is the job of TFS.
Team Foundation Server and Testing Automation
Team Foundation Server contains a feature called team build, or simply build. If you have been reading the previous parts of the Microsoft ALM series, you know that the build feature allows you to not only automate builds of your code, but also to automatically run your (unit) tests. The builds are managed through the Team Explorer window and the Builds node (Figure 5). You can view and queue builds also through the web access portal.
Figure 5. The TFS build functionality can be accessed directly through Team Explorer.
How, then, would you set up an automated unit test run to happen for instance nightly? The philosophy behind the solution is simple: you need to remember that tests in Visual Studio are stored in projects in a similar fashion than regular code files are stored in application projects are.
In addition to unit tests, you can also command Team Foundation Server to run other tests that do not require user intervention. For instance, web tests and load tests are candidates for testing automation. Conversely, user interface testing must still be done by hand, unless they are what are called coded UI tests.
Team Foundation Server is flexible when it comes to scheduling your tests and builds. In addition to the traditional scheduling like daily or every Thursday, you can also instruct Team Foundation Server to start executing tests whenever new code is checked in (Figure 6). You can also specify a gated check-in policy that mandates that code must build before new code is allowed to be stored in the database.
Figure 6. Builds can be started either manually or automatically after a set trigger.
When a Test Fails
Sometimes, not everything goes as planned. This applies to unit tests as well, which can report defects in the tested code. If this happens, what would be a wise way to solve the issue? Perhaps you would log the issue and make sure it gets properly fixed and re-tested. Once again, work items in TFS can prove to be invaluable.
Assume you have a unit test for a calculation method, and the method returns incorrect results. The test summary would indicate that this method failed, and when you view the report, you have the option to create a new work item to record the failure. Since work items can be assigned to someone, you could immediately give the task of fixing the defect to some of the developers in the team. Figure 7 shows an example of unit testing failure. By using the Create Work Item command, you can quickly create a work item that is automatically associated with the test results. When a developer views the work item details, he or she can open the test results and see the technical details about the problem at hand.
Figure 7. After a failed unit test in an automated build, an associated work item can be created.
Since work items can also be associated with
changesets - the actual code files that are in version control - it is easy to keep seamless track of unit test results, work items and even the code associated with the fix. Without such traceability, either much manual documentation work would be needed, or the code fixes would simply be lost in the sea of changes.
Load Testing and Agents
Load testing allows you to simulate a given number of users in a web application or other server-based application. For instance, you could have several client computers sending requests to a web server, and then monitor the performance of the web server and your application.
Load testing helps you find potential bottlenecks (such as inefficient database queries or missing cache hits) in your solution, and gives you an indication of how many users a given software and hardware configuration can sustain. Another possibility is stress testing, in which a high load is placed on an application to see if it fails.
Team Foundation Server and Visual Studio 2010 Ultimate together provide good support for especially web application load testing. The solution consists of one or more client computers running so-called load agents, which a TFS installation commands. When a distributed load testing starts, a test controller communicates the testing parameters to load agents. These load agents will then start sending requests to the server in question. Once the testing has completed, results are collected and shown to the user.
Web application load testing includes many parameters that you can set. For instance, you can specify the number of virtual users, how long a test step is to run, what kind of network connections are available and what browsers are simulated when sending the requests. You can also specify a test mix that allows you to set different probabilities to different test steps. This way, the tests better mimic real users, as they do not mechanically run the same test steps always in the same order (Figure 8).
Figure 8. Load testing allows you to set a test mix useful for example web application testing.
When a test has completed, you can view the results in both numerical and graphical views. Depending on the test type in question, you would either use the Load Test Viewer or the Web Test Viewer. It is also possible to export test results to an Excel worksheet. This gives you even better options in analyzing the data statistically, or comparing the results from one or more test runs to find trends and patterns in the data.
SummaryWhen it comes to application testing, Microsoft Visual Studio 2010 and Team Foundation Server together form a combination that provides something for everyone. Although some of the testing features are provided by Visual Studio alone, using the automation features that TFS provides can take the repetitive work away from team members. The better option is to give computers the job they are best at: automation. Unit testing has become a standard quality insurance tool in development projects. Visual Studio provides good functionality to help developers write and run their tests. Although tests can be run manually within Visual Studio, it is more efficient to schedule them to run automatically for example nightly. This can be done though Team Foundation Server's team build functionality. Load testing is the realm of Visual Studio 2010 Ultimate, and is not available in the other editions like Premium or Professional. With load testing, you can simulate different number of users and create a so-called test mix which allows you to mimic real users within your tests. The test results can be viewed directly for example as graphs, and even exported into Excel for more advanced analytics.
The topics discussed in the article gave an overview of the possibilities Team Foundation Server gives to professional testers. This was the fourth part of the Microsoft ALM series, and the fifth part will focus on team build functionality in more detail and introduce a new feature called Lab Management.
Until next time!