Introduction

Here's a situation the ASP.NET developer is undoubtedly familiar with. The end user says your web application doesn't work and gives an error, possibly the infamous Yellow Screen of Death (Figure 1). But on your development environment, you cannot reproduce the problem. Since you have also disabled detailed error messages on the production server (usually a good practice), the error messages won't give you much information. This is especially the case if you've replaced the regular ASP.NET error messages with your own.

[error.jpg] Figure 1. An ASP.NET error page, or the "Yellow Screen of Death"

Of course, in case of an application failure, it would be beneficial to have some kind of logging in place in your application. Some applications write log entries to text files, databases, and so forth. But these logging capabilities are usually only added after the fact, and then it is already too late. Without ready-made logging in place and with only the web server to help you, what can you do? Let Microsoft IIS 7 give a helping hand.

Learning to Let IIS Give a Helping Hand

Since the earliest versions of IIS (Internet Information Server), the web server has created logs in text-based files. These files can be configured to contain lots of useful information, and can be analyzed with many third- party utilities. But even so, they are quite basic in nature (Figure 2), and only display the result of a request: was it a success or a failure?

Figure 2. A basic text-based log file created by IIS

Of course, if you want to know how often your application(s) fail, then this information can already be useful. By knowing that a certain request URL always causes a 500 Internal Server Error to occur can lead you to the right track and location in code. Thus, if you have nothing else, make sure you start from these log files.

Even though the log files can be useful, they usually are not detailed enough. When the problems with the applications get tougher, you will need better troubleshooting tools. Fortunately, if you are using IIS version 7 or later (version 7.5 is the latest one at this writing), then you have additional methods available. The trick is to learn how to use them. Note that IIS 7 was introduced with Windows Vista in 2006; it is also available on Windows Server 2008, Windows 7 and Windows Server 2008 R2. Alas, you cannot retrofit IIS 7 to older Windows versions, say Windows Server 2003.

IIS 7 contains a new feature called Failed Request Tracing, or FRT for short (Microsoft internally calls the feature as FREB or Failed Request Event Buffering for historical reasons). Failed Request Tracing is an advanced feature that allows you to focus on just what you are after: failed requests. FRT is also versatile; you can set multiple tracing options, and only monitor certain web application types or requests reporting back only certain HTTP status codes. You can also limit monitoring to certain .aspx files. FRT can make a big difference: troubleshooting can take 15 minutes instead of three hours, so with it, there are chances to get home on time.

The following list shows some basic features which you utilize when working with Failed Request Tracing. The list is by no means comprehensive; so flexible is the feature in question.

• Monitor requests to any file type you specify, including .asp, .aspx and .php.
• Check full request and response HTTP headers to make sure all required data is available.
• Calculate how many bytes were sent back and forth, and how long it took to process the request in each pipeline step.
• Filter on requests that take over N seconds to process.
• Filter on certain HTTP response codes, such as 403, 404 or 500.
• Filter on filenames with wildcards, such as "login*.aspx".
• Know which HTTP modules were used to handle the request.
• See how routing and URL rewriting affect request processing.
• Monitor which application pools were used to handle the request.
• See how load balancing (if enabled) is working and which server is responding to the request.
• Select the appropriate providers for tracing events, such as web server caching, compression, ASP.NET infrastructure, and so on.

Even though the name of the feature starts with the word "Failed", you can in fact use FRT to monitor completely valid requests to the web server as well. This way, you can for instance check to see that all HTTP headers are in place and that serving the requests doesn't take too long. In this article, the focus is on monitoring ASP.NET web applications, but as you have already gathered from the above, the feature is useful to investigate with any web application type, including classic ASP, PHP and others.

Installing and Enabling Tracing

To get started with Failed Request Tracing in IIS 7, you must first enable the feature. The following steps are taken from Windows Server 2008, but the instructions would be similar for Windows Vista, Windows 7 and Windows Server 2008 R2.

First, you need to make sure the FRT feature is installed with IIS. By default, IIS 7 only installs those features that are absolutely necessary to prevent security attacks to the server. This means that FRT is also missing by default, so you need to separately install it to be able to use it.

Windows Servers have lately been using the words "role", "feature" and "role services" to describe functionality. Within this terminology, FRT is a role service of the web server role. To install it, start the Server Manager (usually at the top of the Start menu), go to the web server role (available at the tree view on the left), and then choose the "Add Role Services" link from the right (Figure 3).

Figure 3. To add support for tracing into IIS 7, you must add the tracing role service

At this point, a new dialog box opens showing a list of available services for the web server. Under the Health and Monitoring node, make sure Tracing is checked, and then proceed with the Next and Install buttons. After the tracing service has been installed, you should see a message saying "Installation succeeded". Click Close to dismiss the dialog box.

Once you have completed the installation, you can find the Failed Request Tracing Rules icon in the management console (Figure 4). To be able to use FRT, you must enable it for each web site you want to monitor. This is done from the IIS Manager (also available as part of Server Manager) by choosing the site from the left-hand side, and then navigating to the Failed Request Tracing Rules icon.

Figure 4. The FRT icon becomes visible in the management console after installation

Under Actions on the right, click the link "Edit Site Tracing" (Figure 5). This will open a dialog box where you must check the Enable check box, and select the folder to which you want to store the log files. You can also specify how many log files can be created before the older ones start to be deleted.

Figure 5. Tracing must be enabled for each site whose requests you want to monitor

Even though you need to specifically enable FRT for each site, it doesn't mean that every request coming to the site is traced (unless that is what you want to do). Instead, you can configure the feature on different scopes: site-wide, application-wide, and so on. This is important from a maintenance perspective, but also from a performance perspective.

Performance is a consideration because enabling FRT will affect IIS's request processing speed. Therefore, you should only enable the feature when necessary, and then within the appropriate scope. But even if you would enable FRT on a very busy site, the feature will auto-tune itself if there are too many requests in a short period of time. Thus, even gross mistakes in configuring the feature should not bring the web server to a grinding halt.

Configuring Tracing Rules

At this point, Failed Request Tracing has been installed and enabled on your web server and the site of your choice. However, no tracing rules have been configured yet, so this is the logical next step. To configure these rules, return to the IIS Manager, and select the appropriate scope from the tree on the left (for instance, an application). Go to the Failed Request Tracing Rules icon, and double-click it. The middle panel should now display an empty list of rules (Figure 6).

[TracingRules.jpg] Figure 6. By default, no tracing rules are defined

Now at the right-hand side, click the Add link under Actions. A dialog box titled "Add Failed Request Tracing Rule" is now shown (see Figure 7) allowing you to select the type of file (or a part of a filename) you want to trace. Select for instance .aspx, and click Next.

[TracingRules7.jpg] Figure 7. When adding tracing rules, you can specify which request you want to monitor

On the next screen, you are able to specify the conditions under which tracing is done. For instance, you can select that only requests leading to a HTTP status code 500 (Internal Server Error) are traced. Once you have made your choice, click again Next.

The third page on the dialog box allows you to select which tracing providers you want to trace, and the level of detailed information they should create. For ASP.NET web applications, you could check most of the areas under the WWW Server and ASPNET providers. Also, select the appropriate verbosity level for your needs. Once you are done, click Finish to create the rule.

After FRT has been enabled and configured for a particular site, application or file, IIS begins to collect information about matching requests. At this point, use your browser (or the appropriate client application if troubleshooting web services) to access your application to create requests for IIS to trace.

Once request matching tracing rules start to arrive, IIS begins to collect the results. The results are stored in XML files under the specified logging directory (by default C:\inetpub\logs\FailedReqLogFiles). Under this directory, you can find one additional subdirectory with the web site identifier, such as "W3SVC1". The choice of XML over a binary format means that processing the results can be easily automated with your own custom solutions.

The log files themselves are named starting with the letters "fr", and one file will contain the details for a single request. You can double-click any of these XML files and open them in your favorite browser. There, they will render as regular HTML pages with some interactivity in place. Thus, the folder can fill up pretty quickly if you enable tracing on a busy site and have instructed IIS to keep many log files.

Browsing Through the Trace Results

In Figure 8, you can see an example of a trace log file opened in Internet Explorer. Whenever you open these XML files with your browser, an XSL stylesheet transformation is automatically applied, and thus you can view the traces using a convenient HTML interface instead of raw XML data. Remember that you can always open the trace file in your favorite editor (or Notepad) if you want to view the file as it is. Your browser's View Source command should also do the trick.

[TracingResults.jpg] Figure 8. Tracing results opened with a browser

The HTML representation is divided into three separate tabs at the top. Through these tabs you can find more information about the request in question. The main page, "Request Summary" shows you just that: the key points of the request, and possibly the location where the request failed.

The most interesting tab is the "Request Details" tab. This tab shows all the processing steps inside IIS that the request goes through, and finally shows where the processing stopped, in case there were errors (remember, FRT can also trace completely valid, successful requests). In case of a failed ASP.NET web application request, the results would look similar to those in Figure 9.

For instance, assume you had made a mistake in database access logic, and had forgotten to open a database connection:

  string connStr = "Data Source=mysqlserver;Initial Catalog=" +
    "Northwind;Integrated Security=True";
  SqlConnection conn = new SqlConnection(connStr);
  try
  {
    string sql = "SELECT COUNT(*) FROM [customers]";
    SqlCommand cmd = new SqlCommand(sql, conn);
    try
    {
      //conn.Open(); -- Oops!
      int count = (int)cmd.ExecuteScalar();
      Label1.Text = "Record count = " + count;
    }
    finally
    {
      cmd.Dispose();
    }
  }
  finally
  {
    conn.Close();
    conn.Dispose();
  }

If this C# code would execute, an exception would be thrown to indicate that the connection to the database was not yet opened properly. If you would trace such a request with IIS 7's FRT, the details would show something like this:

  139. MODULE_SET_RESPONSE_ERROR_STATUS 
  ModuleName    ManagedPipelineHandler 
  Notification  128 
  HttpStatus    500 
  HttpReason    Internal Server Error 
  HttpSubStatus 0 
  ErrorCode     0 
  ConfigExceptionInfo  
  Notification  EXECUTE_REQUEST_HANDLER

This information can be found from the Request Details tab. This doesn't directly tell information about the .NET exception, but more information can be found from the Compact View tab. Scrolling down to the last few entries, you can find the GENERAL_RESPONSE_ENTITY_BUFFER entry. This often gives tips on where the error occurred inside your application. Of course, other types or errors or misconfigurations show up elsewhere; thus the default tab, Request Summary, is the best place to start.

A Quick Tip About Command-line Management

In addition to the graphical management utilities that IIS provides, you can also work with FRT from the command- line using the AppCmd command (Figure 10). AppCmd can be thought of as a power user tool for administering and maintaining IIS web servers. It is a flexible command, and also supports configuring FRT. And because it is suitable to be called from batch files, you can easily automate repetitive tasks with it.

[AppCmd.jpg] Figure 10. The AppCmd command-line utility allows you to manage IIS

Assuming you have already opened the command prompt and navigated to the correct directory to be able to run AppCmd, you can use commands similar to these to work with FRT.

To enable tracing, run the following command:

  appcmd configure trace "My Web Site" /enablesite

After tracing has been enabled and is running for a while, you can use the following command to list the available tracing files:

  appcmd list trace

For more details about AppCmd's support for FRT, see the command-line help by appending the command name with "/?". Remember that you can directly launch the web browser from the command-line by simply typing the name of the resulting XML trace file, and pressing Enter.

Conclusion

Ironing out problems in web applications requires skill, as web applications are often complex beasts. There are many ways you can try to gather more information from a request, but in production environments, the tools you can use are often limited. With IIS 7's Failed Request Tracing feature, you can get much-needed inner details about application failures, be they web server misconfigurations, application configuration errors, browser/application request failures, or plain old coding bugs. In addition to these, you can also use tracing to monitor quality of service levels (in the form of response times), and also see how the performance of the web server affects your application. With the detailed information you can gather, you can make educated decisions on which parts of the whole system you should optimize.

The Failed Request Tracing feature in IIS 7 is something that all developers and Web server administrators should be aware of. Of course, there are still plenty of Windows based Web servers that don't have IIS 7 yet, but as time marches on, more and more Web servers will have this functionality. It is your job to get the most benefit from it.

About the Author

Jani JC$rvinen is a software development trainer and consultant in Finland. He is a Microsoft C# MVP, a frequent author and has published three books about software development. He is the group leader of a Finnish software development expert group at ITpro.fi and a board member of the Finnish Visual Studio Team System User Group. His blog can be found at http://www.saunalahti.fi/janij/. You can send him mail by clicking on his name at the top of the article.

Resources

IIS web server pages
Learn IIS
"Configuring Tracing for Failed Requests in IIS 7" on Technet