Determine if an Application is Console or GUI

Environment: The attached demo application has been compiled using VC++ 6 (SP 2) on Win NT (SP 4). It has not been tested on Win 95.

This article will discuss a very small utility to check if an EXE file is a console or GUI application. There are no direct API calls, which gives us this information about an application. Whenever an application is created, the EXE file stores a whole lot of information in certain format. Therefore to dig out the information it is important to understand the format of executable files. All the data structure types used for various file headers are defined in WINNT.H file supplied with Win NT OS.

Starting from Win NT 3.1, a new file format was introduced by OS called Portable Executable (PE). The PE file format draws primarily from the COFF (Common Object File Format) specification that is common to UNIX® operating systems. The entire format consists of an MS-DOS MZ header, followed by a Real-Mode Stub Program, the PE file signature, the PE file header, the PE optional header, all of the section headers, and finally, all of the section bodies.

The MS-DOS header occupies the first 64 bytes of the PE file. The contents of this header are represented by IMAGE_DOS_HEADER data structure. The e_magic field of this structure is used to identify an MS-DOS compatible file type. All MS-DOS compatible executable files set this value to 0x54AD, which represents the ASCII characters MZ. MS-DOS headers are sometimes referred to as MZ headers for this reason. The final field, e_lfanew, is a 4-byte offset into the file where the PE file header is located.

The Real-Mode Stub Program is an actual program run by MS-DOS when the executable is loaded. For an actual MS-DOS executable image file, the application begins executing here.

Starting with Windows and OS/2 executables, .EXE files were given file signatures to specify the intended target operating system. For the PE file format in Windows NT, this signature occurs immediately before the PE file header structure.

Following are the signatures defined in WINNT.H file.


#define IMAGE_DOS_SIGNATURE 0x4D5A // MZ

#define IMAGE_OS2_SIGNATURE 0x4E45 // NE

#define IMAGE_OS2_SIGNATURE_LE 0x4C45 // LE

#define IMAGE_NT_SIGNATURE 0x50450000 // PE00

Once we have the location of the file signature, the PE file follows four bytes later. The next 224 bytes in the executable file make up the PE optional header. The information corresponding to optional header is defined by IMAGE_OPTIONAL_HEADER data structure. The optional header contains most of the meaningful information about the executable image, such as initial stack size, program entry point location, preferred base address, operating system version, section alignment information, and so forth. And the information we are looking for i.e. application type. "Subsystem" field identifies the target subsystem for this executable. Each of the possible subsystem values are listed in the WINNT.H file


#define IMAGE_SUBSYSTEM_UNKNOWN 0 // Unknown subsystem.

#define IMAGE_SUBSYSTEM_NATIVE 1 // Image doesn't require a subsystem.

#define IMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Image runs in the Windows GUI subsystem.

#define IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Image runs in the Windows character subsystem.

#define IMAGE_SUBSYSTEM_OS2_CUI 5 // image runs in the OS/2 character subsystem.

#define IMAGE_SUBSYSTEM_POSIX_CUI 7 // image runs in the Posix character subsystem.

#define IMAGE_SUBSYSTEM_NATIVE_WINDOWS 8 // image is a native Win9x driver.

#define IMAGE_SUBSYSTEM_WINDOWS_CE_GUI 9 // Image runs in the Windows CE subsystem.

The strategy to get to IMAGE_OPTIONAL_HEADER data structure has been implemented in OnOK function of the attached dialog based application. To test enter complete path to the EXE file.

For more information on the PE file format, you can read the SDK documentation.

Download demo project - 22 KB

Date Last Updated: April 3, 1999



Comments

  • Available APIs

    Posted by Legacy on 04/04/2003 12:00am

    Originally posted by: Fredric Rylander

    I liked the article. Thank you for sharing.
    
    

    To my knowledge, there are at least two API functions that can help us do the same thing: SHGetFileInfo (shellapi.h) and GetBinaryType (winbase.h). The SDK states that GetBinaryType isn't available on Windows 9x platforms, which is a pity. On the other hand SHGetFileInfo is available on Windows NT 4.0 and Windows 95 or later.

    Although the following call works you might want to consider specifying the address to, and the size of, a SHFILEINFO struct to be fully compatible with the SDK documents.

    DWORD dwExeType = SHGetFileInfo(pszPath, 0, NULL, 0, SHGFI_EXETYPE);

    The return value specifies the type of the executable file:

    * Non-executable file or an error condition.
    0

    * Windows [GUI] application
    LOWORD = NE or PE
    HIWORD = 3.0, 3.5 or 4.0

    * MS-DOS.EXE, .COM or .BAT file:
    LOWORD = MZ
    HIWORD = 0

    * Windows console application
    LOWORD = PE
    HIWORD = 0

    Note:
    Signatures are made up of two ASCII chars read as a word.

    MZ = 0x5A4D - DOS signature.
    NE = 0x454E - OS/2 signature.
    LE = 0x454C - OS/2 LE or VXD signature.
    PE = 0x4550 - Win32/NT signature.

    Note 2:
    Another nice, and very comprehensive, article about the PE file format is written by Matt Pietrek for Inside Windows: "An In-Depth Look into the Win32 Portable Executable File Format" (url at the time of writing: http://msdn.microsoft.com/msdnmag/issues/02/02/PE/default.aspx).


    / Fredric
    (Anti-spam e-mail: fredric dot rylander at home dot se)

    Reply
Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Do you know where your data is? Consumer cloud-based file sharing services store your sensitive company data on servers outside of your control, outside of your policy and regulatory guidelines – maybe even outside your country – and not managed by you. The potential for data leakage, security breaches, and harm to your business is enormous. Download this white paper to learn about file sync and share alternatives that allow you to manage and protect your sensitive data while integrating and …

  • Available On-Demand Today's changing workforce dynamics, economic challenges, and technological advances are placing immense pressure on business leaders to turn their focus on people – their most valuable asset – in order to remain competitive. Research shows that a significant number of new employees quit within one year of taking a new job*. Whether it's through a merger and acquisition, or standard hiring process, like any first impression, early experiences shape their opinions of their new …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds