MFC and .NET: Why ShellExecute Isn’t Dependable and How to Work Around It

Welcome to this week’s installment of .NET Tips & Techniques! Each week, award-winning Architect and Lead Programmer Tom Archer from the Archer Consulting Group demonstrates how to perform a practical .NET programming task using either C# or Managed C++ Extensions.

One of my primary goals in the .NET Tips & Techniques series is to provide practical advice that will help fellow coders save valuable time by offering my experiences as a programmer who’s been using .NET since the very early betas. A prime example of such an experience was when I attempted to slowly migrate a few of my MFC applications to .NET by simply turning on the /CLR option and including the usage of a few choice .NET BCL (Base Class Library) classes. Much to my surprise (and chagrin), a very unlikely function call stopped working completely! That call was to the ShellExecute function. In many of my demo applications, I use ShellExecute from the About Box dialogs in order to allow the user to point his or her browser to the various support pages on my Web site.

The Problem

A bit of basic (process-of-elimination) debugging proved that ShellExecute failed only when the .NET methods were called. When those methods were removed, it worked. After a lot more debugging, I realized that the main issue had to do with the fact that the ShellExecute function needs to be called from an STA (Single-Threaded Apartment) thread when being passed a value that causes the browser to be executed. This is because if the ShellExecute function receives a value starting with “http://”, it uses the Web Browser COM objects to launch the URL. (If you are launching an executable rather than a URL, then the ShellExecute function works correctly.)

This is not an issue with normal MFC applications because by default, the Visual Studio AppWizard generates MFC applications with a call to the AfxOleInit function from the application class’s InitInstance member function. The AfxOleInit function then calls the COM function CoInitializeEx, specifying the STA theading model. However, mixing MFC and .NET brings the .NET CLR (Common Language Runtime) into the picture. When the CLR is invoked, it calls CoInitializeEx and sets the application’s main thread to the MTA (Multi-Threaded Apartment) threading model, which is what ultimately causes the ShellExecute function to fail.

The Solution(s)

There are two solutions to this problem. The first involves performing the following steps in order to ensure that the application’s main thread is an STA thread:

  1. Define a custom manage entry point for your application – /ENTRY:MyMain.
  2. Decorate the new entry point with the [System::STAThread] attribute.
  3. Call the WinMainCRTStartup function from the CRT entry point.

Here’s what that would look like:

extern "C" void WinMainCRTStartup();

#using <System.dll>

[System::STAThread]
void MyMain()
{
  WinMainCRTStartup();
}

However, I personally found it much faster to simply call the System::Diagnostics::Process::Start method. After all, I do have access to the entire .NET BCL when I flip the CLR switch. This is by far the easiest way to solve this problem.

The Ins and Outs of a Workaround

This article was a long-winded way of saying “use the Process::Start method instead of ShellExecute when mixing MFC and .NET code.” However, it’s one thing to know what the workaround is and another to know why you have to perform a workaround in the first place.

So in keeping with the spirit of this series, hopefully the information you read here saves you the several hours I spent tracking all this down.

About the Author

The founder of the Archer Consulting Group (ACG), Tom Archer has been the project lead on three award-winning applications and is a best-selling author of 10 programming books as well as countless magazine and online articles.

More by Author

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Must Read