Extending Visual Studio 6.0 Debugger

Environment: VC6

Everybody has heard about the autoexp.dat file as a way to show custom data types in Visual Studio 6 debugger. It is very useful, but sometimes not satisfactory. For example, if during a debug session you want to see a variable value of some custom data type, e.g. FILETIME. FILETIME is 64-bit value which represents a number of 100 nano second units since January, 1st 1601. This value is divided into a struct containing two 32-bit values: Low and high 32 bits of that 64-bit value.

typedef struct _FILETIME {
  DWORD dwLowDateTime;
  DWORD dwHighDateTime;
} FILETIME, *PFILETIME, *LPFILETIME;

Using documented autoexp.dat capabilities is simply not possible to show date time (and many others custom data types) in a human readable form during a debug session. Until I discovered an undocumented autoexp.dat feature I had used a simple Visual Studio 6.0 add-in which was invoked from my add-in toolbar button. Of course, such solution did the job, but it is a little bit annoying to select a variable or set a mouse cursor to it in the source code window and then press the add-in toolbar button. I thought there is no way to show myself interpreted variable value in a debugger tooltip or watch window.

But, still, there is a way to do it. What you need to do is to write a WIN32 DLL which exports function with the following prototype:

HRESULT WINAPI AddIn_SystemTime( DWORD dwAddress,
                                 DEBUGHELPER *pHelper,
                                 int nBase,
                                 BOOL bUniStrings,
                                 char *pResult,
                                 size_t max,
                                 DWORD reserved )

I will explain the meaning of the parameters and data types soon. After you wrote that function which needs to evaluate your custom data type variable value, you just need to add the following line in your autoexp.dat (AutoExpand section) file:

MyDataType=$ADDIN(MyLibrary.dll,fCustomDataTypeViewer)

where MyDateType is a data type you are writting a custom viewer for, MyLibrary.dll is a dll from which you exported a custom viewer function and fCustomDataTypeViewer is the name of the exported function. It is best to put a full path to the DLL instead of DLL name only, because Visual Studio sometimes gets confused and this feature was not working properly when I put just a dll name without a path.

HRESULT WINAPI FILETIME_Viewer( DWORD dwAddress,
                                     DEBUGHELPER *pHelper,
                                     int nBase,
                                     BOOL bUniStrings,
                                     char *pResult,
                                     size_t max,
                                     DWORD reserved )
{
  FILETIME FileTime;
  SYSTEMTIME SysTime;
  DWORD nGot;
  if (pHelper->dwVersion<0x20000)
  {
     // This is the way to find out which debugger
     // version is running VC 6.0 version
     if (pHelper->ReadDebuggeeMemory(pHelper,
         dwAddress, sizeof(FileTime), &FileTime,
         &nGot)!=S_OK)
             return E_FAIL;

     if (nGot!=sizeof(FileTime))
             return E_FAIL;
  }
  else
  {
     if (pHelper->ReadDebuggeeMemoryEx(pHelper,
         pHelper->GetRealAddress(pHelper),
         sizeof(FileTime), &FileTime,&nGot)!=S_OK)
              return E_FAIL;
     if (nGot!=sizeof(FileTime))
              return E_FAIL;
  }
  // convert to SystemTime
  if (!FileTimeToSystemTime( &FileTime, &SysTime ))
     return E_FAIL;

  // and then just call the simple formatting function
  // which will fill the out buffer

  return FormatDateTime( &SysTime, pResult, max );

}

And now, here is the explanation of types and parameters used:

Each time when your mouse cursor is over some variable (or the variable is in a watch or a quick watch window) of FILETIME data type, the debugger will call your dll's exported function and if it returns S_OK, pResult contents will be shown in a debugger tooltip.

dwAddress is an address of your object in the debugger memory space. Of course, you can't use this address in your custom viewer DLL memory space. You need to read it using a callback function provided with the DEBUGHELPER structure which is declared on the following way:

typedef struct tagDEBUGHELPER
{
  DWORD dwVersion;
  BOOL (WINAPI *ReadDebuggeeMemory)(struct tagDEBUGHELPER *pThis,
                    DWORD dwAddr,
                    DWORD nWant, 
                    VOID* pWhere, 
                    DWORD *nGot );
  // from here only when dwVersion >= 0x20000
  DWORDLONG (WINAPI *GetRealAddress)( struct tagDEBUGHELPER *pThis );
  BOOL (WINAPI *ReadDebuggeeMemoryEx)( struct tagDEBUGHELPER *pThis,
                     DWORDLONG qwAddr,
                     DWORD nWant, 
                     VOID* pWhere, 
                     DWORD *nGot );
  int (WINAPI *GetProcessorType)( struct tagDEBUGHELPER *pThis );
} DEBUGHELPER;

typedef HRESULT (WINAPI *CUSTOMVIEWER)(DWORD dwAddress,
                                       DEBUGHELPER *pHelper,
                                       int nBase,
                                       BOOL bUniStrings,
                                       char *pResult,
                                       size_t max,
                                       DWORD reserved );

For Visual Studio 6.0 debugger version you are interested only in ReadDebuggeeMemory callback. This callback reads the object being debugged in memory into the memory structure provided with your custom viewer. It doesn't necessarily need to be the real object as is in your program. You just need to provide the matching memory layout for the object members that you are interested in.

The next parameter, DWORD nWant, represents a size in bytes you want to copy. VOID* pWhere is a memory where the object/part of the object being debugged will be copied. Finally, DWORD *nGot is an output value that shows how many bytes were copied successfully.

After you read the memory from the program debugged memory space, it is up to you to interpret it and return a result to the debugger. The resulting expression should be copied into char *pResult watching for the maximal buffer size which debugger has sent you through size_t max parameter. Also, there are additional parameters available:

int nBase which is hex or decimal system base of your current debugger setting and BOOL bUniStrings, which should be ignored.

You should be careful, because if your custom evaluator DLL crashes, Visual Studio session will also crash. Not only can one function can be exported from this DLL, but you can export as many functions as many different custom data type viewers you want to introduce.

As I know, this hidden feature is not documented in Visual Studio 6 companion documentation. It appears for Visual Studio .NET, but I tried it also in Visual Studio 6.0 and it works. You even don't need to have any part of Visual Studio .NET installed on your machine.

Of course, this feature is very suitable to evaluate and show some more compound data types, especially classes from your framework. I just gave a single example with a FILETIME object.

Downloads