Application Security Testing: An Integral Part of DevOps
No matter how well developers write and test their applications, application crashes for end-users are still a fact of life. Rather than relying on patchy end-user descriptions of what caused the crash (or, worse still, users silently growing frustrated with the software and switching to a competitor), developers can take advantage of Windows Error Reporting. Entering into an agreement with Microsoft to receive user-provided crash dump files can significantly improve application quality and reduce support costs.
Windows Vista, Windows Server 2008 R2, and Windows 7 provide a range of new functionality for Windows Error Reporting. From an application support perspective, one of the most exciting new features is the Windows Error Reporting Problem Steps Recorder, which ships with all Windows 7 editions. The Problem Steps Recorder executable (PSR.exe) records the steps that a user performs in an application. This article doesn't cover PSR in depth, but you can read this MSDN blog post for more detail on how to use it. PSR promises to significantly simplify a process that currently relies on a combination of screen captures and imprecise user narratives to document hard-to-reproduce crashes.
Using Windows Error Reporting (WER) involves two distinct activities:
- Establishing a commercial relationship with Microsoft to retrieve error reports for applications produced by your organization.
- Using the WER APIs to augment the functionality built into the operating system.
This article focuses primarily on second item: using the WER API.
|Author's Note: For more information on the commercial side, see the Windows Quality Online Services (Winqual) web site and the Windows Error Reporting: Getting Started page. The only financial cost associated with the commercial relationship is the purchase of a VeriSign ID ($399.00 US for a 1-year certificate at this time). Because this is the same ID used for Authenticode, many ISVs will already have a valid ID.|
With the commercial relationship in place, two major features are available to an ISV: the ability to access user-submitted "minidump' files related to applications they have developed and registered for WER reporting through Winqual, and the ability to provide feedback to users about patches, workarounds, available upgrades, and surveys relevant to the application problem. You can load minidump files in any of the Microsoft debuggers (including Visual Studio), which helps developers identify the cause of the problem.
You perform the first level of WER configuration in the Windows Registry. WER settings can be configured at both a per-machine (HKEY_CURRENT_USER\Software\Microsoft\Windows\Windows Error Reporting) and per-user (HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting) level. There are more than twenty possible registry keys you can use to configure WER behaviour. These include the ability to turn WER on and off, control the dialogs shown to users, and specify the location of a corporate WER server that can be used in place of the Microsoft WER server.
For the full list of configuration options see MSDN. One new feature (available with Windows Server 2008 and Windows Vista SP1) is the ability to capture full user-mode dumps locally after a crash. When a minidump file proves to have insufficient information to identify the exact cause of a problem, you can configure a user's machine to capture a full user-mode dump by adding this registry key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\<ApplicationName>.exe. The MSDN Library covers full range of user-mode dump options.
Code Interaction with WER
Given the extensive commercial and configuration aspects of WER, it may seem that there isnt a lot left for the developer to dothe legal department handles the commercial relationship with Microsoft, system administrators configure WER using group policy, and a project manager typically downloads and delivers the crash dump files ready for analysis to developers (of course, for a smaller ISV, legal and project management may be a developer's responsibility). But WER does leave some work for developers. It has a small API set that is particularly relevant to C/ C++ developers, containing over twenty documented functions available in Werapi.h that augment the built-in WER functionality.
WER functions fall into three basic categories:
- WER Basic Control: The WerSetFlags and WerGetFlags functions let you set and retrieve basic WER settings for the calling process. The WerAddExcludedApplication adds a list of opt-out applications to the ExcludedApplications registry key.
- WER Extended Information: For some problems, the minidump files that WER collects will be missing information critical to diagnosing the cause of the problem. As discussed above, a new option available in this case is to configure WER to store a full user-mode dump locally when an error occursbut that option has only become available recently. As an alternative, you can extend the data WER collects by adding memory blocks to the report (achieved via WerRegisterMemoryBlock), adding a specified dump type (WerReportAddDump) or by adding an out-of-process handler responsible for collecting more information to be included in the report (WerRegisterRuntimeExceptionModule).
- Application-Created WER Reports. Beginning with Windows Vista, an application can initiate sending a WER report with WerReportCreate and associated functions. After a report is created, you can call the functions WerReportSetParameter, WerReportAddDump and WerReportAddFile to add data to the report, which can then be sent to the WER server and retrieved in the same way as a standard report. These new Windows Vista functions replace the obsolete ReportFault API that shipped with Windows XP to allow application-initiated reporting.
As developers, our initial reaction when confronted with a problem like capturing data from a users machine to monitor application crashes is to build our own infrastructure to complete the task. It sounds deceptively simplebuild a web service to receive the crash dump file, configure the application to create a dump file each time it crashes, and upload a compressed dump file on application startup if one is detected.
But in addition to these developer-centric tasks, a custom solution requires hosting for the collection web service, the consideration of legal issues related to the collection of potentially sensitive user data, and designing, building, testing, and deploying a patch release notification and delivery mechanism.
Instead, by taking advantage of the inbuilt functionality in WER, you can leverage a trusted and mature error-reporting mechanism to collect field data on application problems. In addition, by taking advantage of newer WER functionality that began appearing in Windows Vista, it's possible to collect a wide range of data that allows full crash investigation.