By Wayne Plourde
If you have a Web site or Web application that makes money, you are probably very interested in who visits your site, how much time they spend, what they look at, and if they come back. Wayne Plourde looks at some of the ways you can track visitors/users on your site with IIS and ASP.NET without spending a fortune on costly log analysis applications.
First, I will begin with a discussion of IIS Web log files and where they fall short in providing a complete picture of your visitor’s activity. Next, I will show you how to build a tracking class to store info about users while they visit. Then, I will explain some options in delivering that data to you. While doing so, I will take advantage of some of the new features of ASP.NET, particularly some new Session and Application context events.
These techniques will be suitable for low to medium traffic Web sites. Hopefully they will give you a quick leg up on understanding traffic patterns on your site before you are forced to purchase a costly Web log analysis program. For a startup or hobby Web site, this may be a prohibitive expense. Of course, if your Web site is on the scale of Amazon.com, the complexity of your customer activity is in a whole different league and the following techniques may be inadequate. In the article, I will anticipate that you have a basic understanding of the Session and Application context and the Request and Response objects, especially how they interact with cookies. In addition, you should have some familiarity with configuring ASP.NET applications with the web.config file.
Finally, we will discuss privacy — what you need to know about the information you collect and the rights of the visitors you collected it from. IIS Web Logs
In case you didn’t know, IIS (Internet Information Server), as with most Web servers, keeps a very thorough record of every request made against it. Each time a page or an image is requested a new record is added to the log file. We know that tracking visitors and customers on our Web sites is important, but what kind of information can we learn from them? People often talk about how many hits they get on their site; however, this isn’t very instructive without some additional information to accompany it.
Who is visiting the site?
Web servers can determine the IP address of each visitor. With this info, we can lookup the “owner” of the IP (usually a service provider) using a WHOIS utility and also run a reverse DNS which can provide a domain name associated with the IP, making it possible to determine the type of organization or even the country from the domain extension, such as .uk (Great Britain) or .nz (New Zealand).
By aggregating the activity on a single IP address, we can approximate the number of unique visitors on the site. I say “approximate” because many Internet users do not have a fixed IP address. Their IP is dynamically allocated by their service provider when they connect. The time a unique visitor spends on the site is called a session. ASP also helps us with this by providing a Session context while the visitor is continually active for a predetermined amount of time. Later, we will discuss how cookies can be used to keep track of visitors across sessions.
Another important piece of data about your visitor is what tool they used to get to your site. Web servers receive a user agent string identifying the application making the request – most often a browser. Sometimes, visitors can be automated tools known as spiders, crawlers or robots. These tools are generally used by search engines to crawl through your site looking for content. If search engines bring customers to your site, then you will want to know how often crawlers, like Googlebot, visit. This knowledge is critical to understanding how fresh your content is on the search engines
What did they look at?
Of course, we not only want to know what pages visitors viewed but also the order in which they were visited. This can tell you where the user entered the site and where they they left. You can also begin to see patterns. For instance, if relatively few visitors are clicking on your product page link from you home page, then it is a pretty good indication that your link is either not prominent enough or your marketing pitch needs to be stronger.
How long did they stay?
It is very useful to know the entire length of the visit in addition to the length of time between page requests. This can tell you if visitors are plowing through the site or spending some time studying the content. It is very important to understand that the HTTP protocol does not maintain a connection to the server between requests, so we must rely on the timestamps of the requests. Unfortunately, because HTTP access does not maintain a connection once the page content is retrieved, it is very difficult to know the amount of time the user looked at the last visited page.
Where did they come from?
When a visitor clicks on a link on another site that links to your site, most browsers pass this information along to the server. This link is known as a Referrer URL. Referrers can be links on other sites, links for search engines or links from ads you have placed. Once you learn to read these, the information can be quite useful. For instance, here is a typical Referrer URL:
From this we can learn several things. First, we know that the visitor came from Google. Second, their search term was “widgets.” Third, the start index for the Google search page was 10. Since the default page size on Google is 10 records, we know our link was on the second page of the search results.
By the way, many webmasters are stating on various forums that over 50% of their traffic is generated by Google – you must understand both the behind the scenes and front end aspects of Google’s operations. Submit your site now for free:
Where Analysis Program fall short
Most analysis tools assume that you are getting mass quantities of visitors, so the data is aggregated and averaged. This means you usually won’t be able to view activity within individual sessions. If you are just starting an eCommerce site, and you are not receiving much traffic, you may want to see exactly what path each visitor took through your site.
In addition, you want to know if visitors return. Log files cannot provide this information, so analysis programs will not be a help either.
Besides, running reports can be a lengthy process. Do you really want to take the time everyday to get some straightforward answers on the activity of your site? There must be an easier way.
What Can ASP.NET Do for Tracking?
If you have control over the code on your ASP.NET site, there is an easier way.
The Session Context
Keep in mind that only browsers support cookies, so when robots visit your site, each request will initiate a new session.
The Session context not only can hold values across page requests, it can also first fire events on Session Start and on Session End. These will be very useful in setting up our tracking object and handling notification and reporting.
The Request Object
The Request object gives us programmatic access to many of the same data items that are stored in the log file such as the requested URL, the Referrer, the UserHostAddress (IP), and the UserAgent. In addition, the object provides a Browser Capabilities object, which can give you very detailed information about what is implied by the user agent string.
Also, cookies are great way to keep track of the activities of registered users. You can store other identifying information about a visitor or user and associate this information with your tracking data. The less anonymous your visitors are, the better able you will be to target particular demographics and eventually convert more visitors to customers. However, if you are storing sensitive data in cookies, you should look at protecting the data with some form of encryption. Cookie data is passed in the clear over non SSL connections.
Adding Comments to the IIS Log
A simple way to provide more information to the IIS log files is to append it yourself with the Response.AppendToLog method. By using Session and Application events you can place keywords that you can later use when search the files. You might even be able to train some of the analysis programs to understand you keywords and hopefully provide more meaningful stats.
Sub Application_BeginRequest(ByVal sender As Object, ByVal e As EventArgs) ‘ Fires at the beginning of each request Response.AppendToLog(“test”) End Sub
There are some restrictions, however. Since the data you provide is appended to the URI Query portion of the log file, you are limited to 80 characters. In addition, you cannot use commas since they are a delimiter for some of the log file formats. Also, if you anticipate that there will be other querystring elements you may want to prepend your string with an ampersand.
Creating a Session Tracker Class in ASP.NET
Now we can finally get to some code. I have created a simple ASP.NET Web application using VB.Net. The application contains six ASP.NET Web forms that hold a menu user control to simplify navigation between the pages. Remember, our purpose to track activity.
Fig 2 – The Home Page
The key element of the example application is the SessionTracker class. The class is designed to assemble all the necessary tracking data on its own and provide that data for reporting through a number of read-only properties:
Pagesan ArrayList of the names and timestamps of the ASP.NET web forms viewed during the sessions
|VisitCountNumber of times the visitor have visited the site|
|OriginalReferrerThe Referrer from the visitor’s first visit|
|OriginalURLThe Requested URL from the visitor’s first visit|
|SessionReferrerThe Referrer for the current visitor session|
|SessionURLThe Requested URL for the current visitor session|
|SessionUserHostAddressThe IP Address of the visitor|
|SessionUserAgentThe Browser or other application|
|BrowserA reference to the HTTPBrowserCapabilites object which provides additional information inferred from the UserAgent|
Initializing the class
In the Session_OnStart method, we create an instance of the class. Dim tracker As SessionTracker = New SessionTracker() Session(“Tracker”) = tracker
When the New constructor method is called as the class is created, we grab an instance of the current HTTPContext with the HTTPContext.Current static method call. This allows us to get access to the Request and Response objects for acquiring the request info and cookies. A reference to the HTTPContext is held as a member variable. Three helper functions are used to deal with the cookie data: incrementVisitCount, setOriginalReferrer, and setOriginalURL. We also set a default expiration time to be used with all our cookies.
Public Class SessionTracker Private _context As HttpContext Private _expires As Date Private _VisitCount As String Private _UserHostAddress As String Private _UserAgent As String Private _OriginalReferrer As String Private _OriginalURL As String Private _SessionReferrer As String Private _SessionURL As String Private _browser As HttpBrowserCapabilities Private _pages As New ArrayList() Public Sub New() ‘HttpContext.Current allows us to gain access to all ‘the intrinsic ASP context objects like Request, Response, Session, etc _context = HttpContext.Current ‘provides a default expiration for cookies _expires = Now.AddYears(1) ‘load up the tracker incrementVisitCount() _UserHostAddress = _context.Request.UserHostAddress.ToString _UserAgent = _context.Request.UserAgent.ToString If Not IsNothing(_context.Request.UrlReferrer) Then ‘set original referrer if not set setOriginalReferrer(_context.Request.UrlReferrer.ToString) _SessionReferrer = _context.Request.UrlReferrer.ToString End If If Not IsNothing(_context.Request.Url) Then ‘set original url if not set setOriginalURL(_context.Request.Url.ToString) _SessionURL = _context.Request.Url.ToString End If ‘set the browser capabilities _browser = _context.Request.Browser End Sub ‘increment the visit count and save in a cookie Public Sub incrementVisitCount() Const KEY = “VisitCount” ‘check is cookie has been set yet If IsNothing(_context.Request.Cookies.Get(KEY)) Then _VisitCount = 1 Else _VisitCount = _context.Request.Cookies.Get(KEY).Value + 1 End If ‘set or reset the cookie addCookie(KEY, _VisitCount) End Sub ‘set the original referrer to a cookie Public Sub setOriginalReferrer(ByVal val As String) Const KEY = “OriginalReferrer” ‘check is cookie has been set yet If Not IsNothing(_context.Request.Cookies.Get(KEY)) Then _OriginalReferrer = _context.Request.Cookies.Get(KEY).Value Else addCookie(KEY, val) _OriginalReferrer = val End If End Sub ‘set the original url to a cookie Public Sub setOriginalURL(ByVal val As String) Const KEY = “OriginalURL” ‘check is cookie has been set yet If Not IsNothing(_context.Request.Cookies.Get(KEY)) Then _OriginalURL = _context.Request.Cookies.Get(KEY).Value Else addCookie(KEY, val) _OriginalURL = val End If End Sub ‘add the page to an arraylist in the session Public Sub addPage(ByVal pageName As String) ‘create a new page tracker item Dim pti As New SessionTrackerPage() pti.PageName = pageName ‘set a time stamp pti.Time = Now ‘add the page tracker item to the array list _pages.Add(pti) End Sub Private Sub addCookie(ByVal key As String, ByVal value As String) Dim cookie As HttpCookie cookie = New HttpCookie(key, value) cookie.Expires = _expires _context.Response.Cookies.Set(cookie) End Sub #Region “Properties” ‘Visit Count ReadOnly Property VisitCount() As Integer Get Return _VisitCount End Get End Property ‘Original Referrer ReadOnly Property OriginalReferrer() As String Get Return _OriginalReferrer End Get End Property ‘Original URL ReadOnly Property OriginalURL() As String Get Return _OriginalURL End Get End Property ‘Session Referrer ReadOnly Property SessionReferrer() As String Get Return _SessionReferrer End Get End Property ‘Session URL ReadOnly Property SessionURL() As String Get Return _SessionURL End Get End Property ‘Session User Host Address (IP) ReadOnly Property SessionUserHostAddress() As String Get Return _UserHostAddress End Get End Property ‘Session User Agent ReadOnly Property SessionUserAgent() As String Get Return _UserAgent End Get End Property ‘Pages – array list ReadOnly Property Pages() As ArrayList Get Return _pages End Get End Property ‘Browser Cap ReadOnly Property Browser() As HttpBrowserCapabilities Get Return _browser End Get End Property #End Region End Class
Storing the SessionTracker in the Session context
Being able to create a class like the SessionTracker and store it in the Session context is one of the great advantages of ASP.NET. Of course, in ASP 3.0 we could have easily created a similar COM object; however, if it was developed with VB, we could not have saved it in the Session. In case you were not aware of the problems in saving Single Threaded Apartment (STA) components in a session or application context, read the following.
Privacy policies are especially important if your site is geared towards children. In this case, it falls under the jurisdiction of the Children’s Online Privacy Protection Act, which has very strict guidelines about what you can and can’t do with information you collect on your site. Here is an excellent article that gives an overview of the subject:
Controlling privacy in Internet Explorer 6
Browsers are now stepping in and providing tools to alert you to privacy issues that you might not normally be aware of. Here is an overview of the many privacy features in Internet Explorer 6.
Platform for Privacy Preferences
There are also a number of free P3P editors
I have created both a privacy.htm and a privacy.xml file using the Alpha works editor and have included then into the sample web app. Once you have created the files, just link the HTML page to your home page and target the xml file in the head of each of your content pages as follows:
<meta http-equiv=”P3P” content=’policyref=”privacy.xml”‘> <link rel=”P3Pv1″ href=”privacy.xml” type=”text/xml”> For a broad overview of P3P issues, read the following: http://html.about.com/library/weekly/aa040802a.htm
Deploying the Sample Project
There are just a few things you need to do.
- Create a new virtual directory in the Internet Services Manager
- Make sure the application has been created under the new Virtual Directory’s properties dialog.
- Expand the zip file into the directory
- Open the solution file in Visual Studio .Net to build and run the app.
The more you know about your potential customers the better able you will be to convert them into paying customers. Now there is no excuse not to know who is coming to your site.
About the Author
Wayne Plourde is a consulting Software Architect who began his career as a building architect twenty years ago. In 1995, he succumbed to the call of the World Wide Web, and since then has been designing sophisticated Web-based and client-server applications for corporations around the country. Wayne holds both MCSD and SCJP certifications and is one step away from a .NET MCAD certification. You can contact Wayne at his Web site at http://www.plourdenet.com.