Your Free Search Engine: Microsoft Indexing Server

Introduction

Many Web applications provide a search capability that allows users to search all the content of a Web application. This seems at first glance to be a difficult task. How can you search all the files in your Web application and at the same provide robust search capabilities most users have come to take for granted? For example, you can search for any combination of words with AND or OR combination, search for phrases, or search for partial word matches. Building such a search engine is not a small task. There are many custom solutions out there that create their own search algorithms. They are either cheap but provide only limited search capabilities or they are expensive.

But, there is no need to build your own solution or to buy an expensive one. Microsoft provides a solution to this problem: Microsoft Indexing Server. The Microsoft Indexing Server comes as part of Windows 2000, Windows XP, and Windows 2003 and does not require any additional licensing. This article will explain the capabilities of the Indexing Server. It will also walk through the code about how you can use the Indexing Server to provide search capabilities right from within your application. It is important to understand that the Indexing Server indexes files on the files system. It is not a Web spider that can walk your Web site and find all linked pages and index them. Web spiders are used by the search engines such as Google, MSN, Yahoo, and so forth to index Web pages. But, Indexing Server can be pointed to a local Web site, get the physical path where the files for this Web site are located, and then index those files and also store the virtual path of those files. This way, you can still index the files of your local Web application and know the URL for each file.

How to Install Microsoft Indexing Server

Windows 2000 comes with MS Indexing Server 2.0, and Windows XP and Windows 2003 come with MS Indexing Server 3.0. This article will concentrate on MS Indexing Server 3.0, although almost everything applies to the previous version. MS Indexing Server is a separate Windows component that needs to be installed. Go to "Add or Remove Programs" in your Control Panel and select "Add/Remove Windows Components." Make sure that the component "Indexing Service" is installed. It is important to install this component after IIS has been installed. IIS has an Indexing Service extension, which will only get installed if the Indexing Service is installed after IIS. If this component does not show up, then uninstall the "Indexing Service" component and then reinstall it again. Afterwards, you will see the "Indexing Service" extension also in IIS. More info to this issue can be found here.

How to Start the Indexing Service

You can configure the Indexing Server through the "Computer Management." You find under the entry "Services and Applications" in the left side pane an entry called "Indexing Service." Right-click on "Indexing Service" and select Start from the popup menu. The first time you do this, it will ask you whether you want to start the service when the computer starts up. Answer with yes, which will set the "Indexing Service" to automatic startup. Through here you can also stop, pause, and resume the Indexing Service. When you expand the "Indexing Service" entry with the plus sign then you can see a list of catalogs. A catalog is a group of folders which gets indexed for search. The Indexing Server comes out of the box with two catalogs, but you can create custom catalogs as needed.

How the System Catalog Is Used

The System catalog is used by the Windows file search function. The Windows file search is opened when you select "Search" from your Windows start menu or open up the file explorer and click the "Search" button. If the Indexing Server is not running or if there is no System catalog then this will search the actual file system. But, it will utilize the System catalog when available and when the Indexing Service is running. You also can configure the Search function to utilize the System catalog or not. At the bottom of the search pane (in the file explorer), select the option "Change Option." Select the option "With Indexing Service" and then select the option "Yes, enable Indexing Service." You can the same way turn the usage of the "Indexing Service" off. Utilizing the Indexing Service will make searching faster as all the files have been already indexed and the system just needs to search the index instead of the file system itself.

Expand the "System" catalog with the plus sign to find out more details about it. Select the Directories entry to see all the folders that are included or excluded in this catalog. By default, this includes the folders "c:\" and "c:\Documents and Settings" but it excludes the folders "c:\Documents and Settings\*\Application Data\*" and "c:\Documents and Settings\*\Local Settings\*." This means it will exclude the "Application Data" folder with all its subfolders as well as the "Local Settings" folder and all it sub folders for any user profile. You can double-click each folder and change it to include or exclude. You also can add new folders by right-clicking the "Directories" entry in the left side pane and then selecting "New | Directory" from the popup menu. You can enter a local folder or a UNC path to any network folder. For network folders, you also need to enter the username and password to use.

How the Web Catalog Is Used

The Web catalog is available only when IIS has been installed. It by default points to the "Default Web Site" of the local IIS instance. Right-click on the "Web" entry in the left side pane and select Properties from the popup menu. Select the Tracking tab to see which Web site this catalog is pointing to. By default, this is the "Default Web Site." You can select any existing Web site of your local IIS instance. The Indexing Service will find the physical path of this Web site and add it to the directories to be indexed. It will also look for all virtual folders configured under this Web site and again add their physical paths to this catalog.

The "Indexing Service" extension ties IIS right into the Indexing Server. In the IIS Manager, open the properties dialog box of the Web site or any virtual folder configured under this Web site. Under the "Home Directory" or "Virtual Directory" tab, you see the option "Index this resource." Unselecting this option and save the settings to remove this physical path from the appropriate catalog in the Indexing Server (this requires that the catalog is started and may take a bit till it makes that change). Selecting this option will automatically add the physical folder again for the appropriate catalog. This allows you to control, right from within the IIS Manager, which folders will be indexed. You also have the "Index this resource" option for all file system folders shown under a Web site or virtual folder. But, I have not seen that this option makes any difference for file folders shown in the IIS Manager.

For this to work on IIS 6 that comes with Windows XP and Windows 2003, you need to make sure that the "Indexing Service" extension is running. Open the IIS Manager and open the "Web Service Extension" item in the left side pane. On the right side, it shows all extensions and, by default, the "Indexing Service" is prohibited. Select it and enable it through the "Allow" button. Also, make sure that IIS has been installed before the Indexing Service as explained earlier.

Other Administrative Options that Are Available

You can create new catalogs by right-clicking on the "Indexing Service" entry in the left side pane and selecting "New | Catalog" from the popup menu. Enter the name of the catalog and the folder where the catalog files are stored. You then need to stop and restart the Indexing Service itself so that the catalog files for this new catalog are created. You can also stop, pause, and start individual catalogs by right-clicking on the catalog name and selecting the appropriate option under "All Tasks" in the popup menu. You can add or remove folders to be included through the "Directories" entry under the catalog name. If you want to index a Web site, open up the properties of the catalog (right-click on the catalog name and select Properties from the popup menu) and, under the "Tracking" tab, select the Web site to index.

The Indexing Service also can create an abstract for the indexed files. Select the "Generation" tab of the properties dialog and uncheck the "Inherit above settings from Service" option. Then, select the "Generate abstracts" option and enter the maximum length of the abstract. You also can set this through the properties of the Indexing Service itself, which then applies to any catalog that inherits the settings from the Indexing Service.

Windows also gives administrators control over which folders or files can be indexed by the Indexing Server. This allows you to protect sensitive files so that they never get included in an index and therefore will never show up in a search result. A good example would be any financial details about the company. Open the file explorer and navigate to the appropriate folder or file. Bring up the properties of the folder or file, click on the Advanced button, and then uncheck or check the "For fast searching, allow Indexing Service to index this file" option, and then save the settings. If you selected a folder, it will ask you whether this setting should be applied to all subfolders or just the selected folder itself. For example, the actual catalog files themselves have all unchecked that setting so that Indexing Server will never try to index its own catalog files.

You also can query the catalog through the Computer Management console. Expand a catalog with the plus sign and you will see an entry called "Query the Catalog." This brings up a Web page with a simple query form. You can perform simple or advanced searches. Select the "Standard query (free text)" option, type in a search term, and then click the Search button. This will perform a simple search and display any matches below the search form. Select "Advanced query" and then type in a complex search term that can include operators such as AND and OR. You will look at the actual query language used by Indexing Server later in this article. This is a convenient way for administrators to test the actual catalog.

Indexing Server is very easy to use and very powerful. The only annoying things for a production usage are the fact that you need to restart the Indexing Service after creating a new catalog as well as stopping the Indexing Service before deleting a catalog. Sure, these are not everyday tasks, but when used with many catalogs—potentially for many customers in a hosted environment—this is a bit annoying.

Your Free Search Engine: Microsoft Indexing Server

How to Query an Indexing Server Catalog from Within Your Code

You can query an Indexing Server catalog through the standard OLEDB data provider. The connection string tells the OLEDB data provider which provider is used, which in this case is the MSIDXS (Microsoft Indexing Server) provider. You can query a local Indexing Server catalog or a catalog on a remote Indexing Server. When querying a local Indexing Server catalog, you also specify the data source in the connection string. Use the name of the local catalog as the data source, for example:

string ConnectionString  = "Provider=MSIDXS; Data Source=\"Web\";";
const string QueryString = "SELECT * FROM WEBINFO;";

You don't specify a data source when you query a catalog on a remote Indexing Server. So, you just specify the provider. The name of the remote Indexing Server and the catalog are then specified in the query itself. Here is an example:

const string ConnectionString = "Provider=MSIDXS;";
const string QueryString = "SELECT *
   FROM EnterpriseMinds.Web..WEBINFO;";

The following sections will cover the query language in more detail. But, you can see that the fact that the FROM clause has changed. First, you specify the name of the remote machine followed by a dot, then the Indexing Server catalog you want to query on that remote machine, followed by two dots, and finally the actual view you want to query. The following code snippet shows how to connect to the OLEDB data source, execute the query, and return a data reader with the result-set.

public static IDataReader Query(string ConnectionString,
                                string QueryString)
{
   // get a OLEDB connection object and set the connection string
   OleDbConnection Connection = new OleDbConnection();
   Connection.ConnectionString = ConnectionString;

   // set the query string to execute
   IDbCommand Command = Connection.CreateCommand();
   Command.CommandText = QueryString;
   Command.CommandType = CommandType.Text;

   // open the data connection
   Connection.Open();

   // Execute the query and return the data reader; when it gets
   // closed, it also closes the connection object
   return Command.ExecuteReader();
}

The caller needs to provide the connection string and the query string. First, you create an OLEDB connection object and set the connection string on it. Next, you create an OLEDB command object and set the query string. Finally, you open the OLEDB connection, execute the command, and return an OLEDB data reader. When the caller closes the OLEDB data reader, it also will close the underlying OLEDB connection. In the next code snippet, you utilize the above method to provide a simplified LocalQuery method. The caller passes along the catalog name and query string and does not need to know the details of the connection string:

// - the OLEDB data provider to use for searching the Indexing Server
//   is MSIDXS;
// - when we connect to a specific indexing catalog, you specify
//   the datasource part of the connection string and set it to the
//   indexing catalog name
const string ProviderConnectionString   = "Provider=MSIDXS;";
const string DataSourceConnectionString = " Data Source=\"{0}\";";

public static IDataReader LocalQuery(string CatalogName,
                                     string QueryString)
{
   string ConnectionString = ProviderConnectionString +
      String.Format(DataSourceConnectionString, CatalogName);

   // perform the query and return result
   return Query(ConnectionString, QueryString);
}

The method assembles the connection string, which also includes the catalog name as data source and then calls the Query method passing along the connection string and the query string. The next code snippet again uses the above Query method to create a RemoteQuery method. It allows you to query any catalog on any remote machine available. It again hides the details of the connection string and how to modify the query string itself:

// - the OLEDB data provider to use for searching the Indexing
//   Server is MSIDXS;
const string ProviderConnectionString = "Provider=MSIDXS;";

// form clause and dot-notation character
const string FromClause  = " FROM ";
const string DotNotation = ".";

public static IDataReader RemoteQuery(string RemoteMachineName,
                                      string CatalogName,
                                      string QueryString)
{
   // replace the FORM clause with a FROM
   // remote_machine.catalog_name..from_clause, for example
   // FROM SCOPE() against the remote machine enterpriseminds and
   // the catalog WEB becomes FROM enterpriseminds.web..SCOPE();
   // this allows to query remote indexing catalogs
   QueryString = QueryString.Replace(FromClause, FromClause +
                                     RemoteMachineName +
                                     DotNotation +
                                     CatalogName +
                                     DotNotation +
                                     DotNotation);

   // perform the query and return result
   return Query(ProviderConnectionString, QueryString);
}

The method searches for the FROM clause in the query string and then replaces it with FROM followed by the remote machine name, a dot, the catalog name, and two dots. It uses a connection string without the data provider. Finally, it again calls the Query method, passing along the connection string and the modified query so it can query the remote Indexing Server catalog. As you can see, it is very easy to query local and remote catalog using the OLEDB data provider.

Your Free Search Engine: Microsoft Indexing Server

Which Query Language Is Used by the OLEDB Indexing Server Provider

The MSIDXS provider supports the SQL query language that is well known and makes it very easy to query the Indexing Server. It only allows you to query, so does not support any updates, inserts, or deletes. The supported query syntax is slightly adapted for this provider. You can find detailed documentation here. The biggest difference is in the FROM clause. You can specify an existing view or the SCOPE() function. The Indexing Serve comes with the following pre-defined views. The referenced MSDN help page lists which fields are included in which view. You have, for example, used the WEBINFO view in the above query string. You are allowed to use a "SELECT * FROM view", which you are not allowed to do when using the SCOPE() function. You also can create your own views by using the CREATE VIEW command. Here is the CREATE VIEW command used to define the WEBINFO view:

CREATE VIEW WEBINFO AS
   SELECT Vpath, path, FileName, size, write, attrib,
          Characterization, DocTitle
   FROM SCOPE()

Instead of a view, you can use the SCOPE() function that defines the scope of the select. The SCOPE() function supports the following arguments:

DEEP TRAVERSAL OF—This searches the paths specified and all the folders beneath it. For example, DEEP TRAVERSAL OF "/" searches the root folder and all the files and folders underneath it. This quite actually is the default scope used when you only specify SCOPE(), which means everything on the Web site is included in the search.

SHALLOW TRAVERSAL OF—This searches the specified paths only, meaning it excludes any subfolder there might be. For example, SHALLOW TRAVERSAL OF "/" searches only the root folder of the web site but none of the sub-folders.

You can list as many paths as you want for both DEEP TRAVERSAL OF and SHALLOW TRAVERSAL OF. You also can include each command multiple times. Here are a few examples to look at:

FROM SCOPE(' SHALLOW TRAVERSAL OF ("/", "/Help") ')—This includes only the root and Help folder in the search. None of the sub-folders are included.

FROM SCOPE(' "/Info", "/Help" ')—The DEEP_TRAVERSAL OF argument is the default argument. This example includes the Info and Help folders and all their sub-folders.

FROM SCOPE(' "/Info" ', ' SHALLOW TRAVERSAL OF "/Help" ')—This example performs a DEEP TRAVERSAL on the Info folder, which means it includes all sub-folders and only includes the Help folder without its sub-folders.

The WHERE clause supports the same filtering as the standard SQL language. Please refer to this article for a complete description. The LIKE operator can be used to perform pattern matching with wildcard characters. The MATCHES operator can be used to perform pattern matching with regular expressions. The FREETEXT operator performs a best matching of words and phrases. And, the CONTAINS operator performs text matching, including proximity searches and stemming. Here are a few examples:

WHERE FileName LIKE '%web%'—Returns all items where the file name contains the word web.

WHERE FileName LIKE '%config'—Returns all the items where the file name ends with the word config.

WHERE CONTAINS(FileName, ' "web" OR "default" ')—Returns all the items where the file name includes the word web or default.

WHERE FREETEXT(FileName, ' default web ')—Returns again all the items where the file name includes the word web or default.

WHERE MATCHES(Contents, ' |(product|)|{2,3|} ')—Returns all the items where the contents includes the word product two to three times.

This allows you to build some very complex queries utilizing the standard SQL language. Building your own search engine that supports all this would be a mayor undertaking. The following article lists and explains all the fields you can utilize in your SELECT statements.

Which Files the Indexing Server Is Capable of Indexing

You have seen so far how to set up Indexing Server to index files, folders, and Web sites. You also covered how you can connect to Indexing Server catalogs and programmatically search them using the OLEDB data provider and the SQL query language. What you have not yet covered is what types of content Indexing Server searches for you and how can you extend this. The Indexing Server extracts the content of files by using so-called filters. Filters are components that implement the IFilter interface and do understand how to read the content of a file. The content of an HTML file is different than that of a RTF file, Microsoft Office file, or a plain text file. Therefore, there are different filters available for these different file types. The "MSN Desktop Search" utilizes the same filter plug-ins. The following article lists which filter is able to read which file types. It also lists a number of additional available filters and from where you can download them. Keep in mind that the article talks about the MSN Desktop Search, but all these filters apply also to the Indexing Server.

All Registry settings for the Indexing Server can be found at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\ContentIndex. You see, for example, a sub key called Catalogs that lists all the catalogs defined on this Indexing Server. For each catalog, you also can find all the settings. Under the ContentIndex key, you find a number of settings for the Indexing Service; one of them is called DLLsToRegister. It lists all the DLLs that Indexing Server will register when starting up. This also includes also all the filters used by Indexing Server. Comparing this with the article that lists all the available filters, you see that out-of-the-box the following filters are installed:

  • query.dll—Filters files with the TXT, ASM, BAT, C, CPP, CXX, CMD, DEF, DIC, H, HPP, and XML extensions. These are all read as plain text files.
  • nlhtml.dll—Filters files with the ASCX, ASP, ASPX, CSS, HHC, HTA, HTM, HTML, HHT, HTW, HTX, ODC, and STM extensions. These are all files that contain or render HTML content.
  • offfile.dll—Filters files with the DOC, DOT, POT, PPS, PPT, XLB, XLC, XLS, and XLT extensions. All these files are MS Office files.
  • mimefilt.dll—Filters files with EML extension, which are MIME content.
  • mspfilt.dll—Filters files with the TIFF extension. This filter gets installed by MS Office 2003.

You can find a complete description of all the Indexing Server Registry keys here. If you are interested in writing your own filter for your custom file types, follow this link. It provides a complete description of the IFilter interface you need to implement.

Summary

Microsoft Indexing Server is a powerful indexing and search engine for your Web or file search. The OLEDB data provider in conjunction with the SQL query language makes it very easy to query the index created by the Indexing Server. With no additional effort, you can provide a simple text search as well as a powerful search with partial word, word, and phrase matching. This includes stemming as well as language sensitive searching. This means there is no additional effort on your side, regardless if you index and search English or Asian content. The filter framework provides the ability to extend the Indexing Server to search any custom content you might have. It is also very nice that you can use the same filters for the MSN Desktop Search engine.

The attached sample application allows you to search any local or remote Indexing Server catalog. You can enter the SQL query string to execute or select from a set of default query strings. The result-set is shown in a list view. It also shows the number of returned matches. If you have comments on this article or this topic, please contact me @ klaus_salchner@hotmail.com. I want to hear if you learned something new. Contact me if you have questions about this topic or article.

About the Author

Klaus Salchner has worked for 14 years in the industry, nine years in Europe and another five years in North America. As a Senior Enterprise Architect with solid experience in enterprise software development, Klaus spends considerable time on performance, scalability, availability, maintainability, globalization/localization, and security. The projects he has been involved in are used by more than a million users in 50 countries on three continents.

Klaus calls Vancouver, British Columbia his home at the moment. His next big goal is running the New York marathon in 2006. Klaus is interested in guest speaking opportunities or as an author for .NET magazines or Web sites. He can be contacted at klaus_salchner@hotmail.com or http://www.enterprise-minds.com.

Enterprise application architecture and design consulting services are available. If you want to hear more about it, contact me! Involve me in your projects and I will make a difference for you. Contact me if you have an idea for an article or research project. Also contact me if you want to co-author an article or join future research projects!



Downloads

Comments

  • There are no comments yet. Be the first to comment!

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Packaged application development teams frequently operate with limited testing environments due to time and labor constraints. By virtualizing the entire application stack, packaged application development teams can deliver business results faster, at higher quality, and with lower risk.

  • Do you know where your data is? Consumer cloud-based file sharing services store your sensitive company data on servers outside of your control, outside of your policy and regulatory guidelines – maybe even outside your country – and not managed by you. The potential for data leakage, security breaches, and harm to your business is enormous. Download this white paper to learn about file sync and share alternatives that allow you to manage and protect your sensitive data while integrating and …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds