Using the Console Like MSDEV

Environment: VC6 SP4, NT4 SP6, 2000 SP1

Probably everybody who reads this article is familiar with msdev.exe. We know that if we type "msdev /?", we get usage information on the console. However, if we type "msdev", the IDE comes up. This dual behavior is achieved through an additional executable, called msdev.com. In this article you can find a fairly generic solution for adding the same capability to any kind of non-console program.


Click here for larger image

In the demo there are two executables, tester.com and tester.exe. By placing them into the same directory and running tester.com from the command line, tester.exe will be spawned. Tester.exe is a simple MFC SDI application which has the extra capability of being able to read/write from/to standard output/error/input of tester.com by using either cin/cout/cerr or printf/scanf.

To use this functionality in your own non-console application, you have to
- call one function (see below) before making any calls that use the console and
- you will need to rename tester.com to "NAME_OF_YOUR_EXE.com".

Then, when you type NAME_OF_YOUR_EXE, the .com executable will start, create the framework needed to enable its console for the .exe and spawn the process. Here is how it works: Tester.com first creates the commandline that will be used to spawn the .exe process. It is done by replacing the .com extension with .exe and appending the rest of the command line arguments. Then, using the constructed command line, the new process is spawned in suspended mode. Then we create three named pipes, one for stdin, one for stdout and one for stderr. Three thread functions will serve as handlers on each pipe. The example below shows what the stdout pipe handler does:

void 
OutPipeTh(void* param)
{
 TCHAR buffer[1024];
 DWORD count = 0;

 ConnectNamedPipe(coutPipe, NULL);

 while(ReadFile(coutPipe, buffer, 1024, &count, NULL))
 {
  buffer[count] = 0;
  cout << buffer << flush;
 }
}
These three thread functions are spawned in their own thread and then the suspended process is resumed. While the .exe application is running, the communication between the .com and .exe on the .com side is handled by these three threads. When we detect that the .exe application exited, we close the named pipes a return the exit code that we retrived from .exe process.

The .exe program needs to call a function at the beginning of its execution in order to establish connection with the pipes created by the .com application. The function is as follows:

BOOL InitializeDualMode(BOOL initMode)
{
 BOOL rc = FALSE;
 TCHAR szOutputPipeName[256];
 TCHAR szInputPipeName[256];
 TCHAR szErrorPipeName[256];

 // construct named pipe names
 //
 _stprintf( szOutputPipeName, 
 _T("\\\\.\\pipe\\%dcout"),
 GetCurrentProcessId() );

 _stprintf( szInputPipeName, 
 _T("\\\\.\\pipe\\%dcin"),
 GetCurrentProcessId() );

 _stprintf( szErrorPipeName, 
 _T("\\\\.\\pipe\\%dcerr"),
 GetCurrentProcessId() );

 // attach named pipes to stdin/stdout/stderr
 //
 rc = _tfreopen( szOutputPipeName, "a", stdout ) != NULL &&
 _tfreopen( szInputPipeName, "r", stdin ) != NULL &&
 _tfreopen( szErrorPipeName, "a", stderr ) != NULL;


 // if unsuccessful, i.e. no console was available
 // and initmode specifiec that we need to create one
 // we do so
 //
 if ( !rc && initMode)
 {
  rc = AllocConsole();
  if (rc)
  rc = _tfreopen( _T("CONOUT$"), "a", stdout ) != NULL &&
  _tfreopen( _T("CONIN$"), "r", stdin ) != NULL &&
  _tfreopen( _T("CONERR$"), "a", stderr ) != NULL; 
 }

 // synchronize iostreams with standard io
 //
 if ( rc )
  ios::sync_with_stdio();

 return rc;
}
As seen in the function, connecting stdio to pipes is painless. One thing to remember is that the pipes in the .com program have to be created with pipemode set to PIPE_TYPE_BYTE | PIPE_READMODE_BYTE. Trting to use PIPE_TYPE_MESSAGE | PIPE_READMODE_MESSAGE will result in communication failure, because the _tfreopen() will try to connect to the pipes in BYTE mode.

Acknowledgements

Thanks to Zoltan Csizmadia for the idea!

Downloads

Download source - 20 Kb
Download executables - 10 Kb