Subsystem Switch

Environment: VC6, SP5, 95/98/NT/Win2K

This simple program changes the program's subsystem attribute.

It seems to be working, but don't trust it completely. If you want to test some files, please back them up first.

One day, for some reason I wanted to make a Win32 console program that doesn't pop up the console windows. I tried many ways and I found it! Linker options (/SUBSYSTEM:Windows /ENTRY:main) can do that, but I wanted to change the already compiled program's subsystem, so I examined the binary file and found it.

Probably, Windows (OS) checks the PE (Portable Executable) file's IMAGE_OPTIONAL_HEADER.Subsystem. There are some other values, but only we need IMAGE_SUBSYTEM_WINDOWS_GUI(2) or IMAGE_SUBSYSTEM_WINDOWS_CUI(3). If the value is ..._GUI, console windows doesn't pop up; but, if the value is ..._CUI, console windows pops up.

The following picture shows one PE file's hex dump. The subsystem's value is highlighted (3, so CUI) but doesn't have absolute offset.

Where to Use this Code

If you change console -> gui, you can get away from the console window, and all console I/O functions (ex:printf) would fail. If you change gui -> console, you can use console I/O functions for debugging. Although you may need this rarely, generally the console program's size is smaller than the GUI with its default setting.

You can find the PE specs in MSDN's Sepcifications Section. WINNT.H contains IMAGE_SUBSYSTEM_WINDOWS_GUI, IMAGE_SUBSYSTEM_WINDOWS_CUI.

Miscellany

The Linker options (/SUBSYSTEM:Windows /ENTRY:main) method may have been on codeguru already, but I didn't know. Oops! It could be a waste of labor, but it's still helpful. When I found the solution, SOMEONE said why don't you post it on codeguru. So, I looked for how to (this the first time for me) and where to post, and I found that. (Thanks to JaeJun Oh for some inspiration. And he is the SOMEONE.)

Downloads

Download demo project - 11.8 Kb
Download source - 5.32 Kb


Comments

  • Usefull demo

    Posted by Legacy on 09/10/2003 12:00am

    Originally posted by: Barbro Sagerholm

    Thanks!
    Solved my problem (I think), how to write a console
    program in VB6. I used the ReadConsole and WriteConsole functions from WinAPI "Kernel32" and then transformed the
    VB6 GUI EXE-code to a console application with your Demo!

    Reply
  • Good sample!

    Posted by Legacy on 08/05/2003 12:00am

    Originally posted by: tchouch

    Very helpfull, but there are some little bugs:
    in case IDC_BUTTON_OPEN:
    1. change : if ( hFile )
    to: if ( hFile != INVALID_HANDLE_VALUE)
    2. FILE_SHARE_WRITE option may be dangerous;
    3. change: if( hFile == NULL )
    to: if( hFile == INVALID_HANDLE_VALUE)

    All hFile = NULL and if(hFile) etc. should be changed to
    hFile = INVALID_HANDLE_VALUE or if(hFile != INVALID_HANDLE_VALUE ) too.

    Reply
  • Nice code.

    Posted by Legacy on 06/01/2002 12:00am

    Originally posted by: Dinu Scobici

    Nice code. Thanx.

    Reply
  • use EDITBIN

    Posted by Legacy on 05/28/2002 12:00am

    Originally posted by: that

    The tool EDITBIN, which comes with MSVC, can change this and other flags.

    Reply
Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • On-demand Event Event Date: August 27, 2015 With the cloud enabling companies to spin up servers and stand up data stores more quickly, mobile apps can be created faster, reducing the time-to-value. But three major obstacles stand in the way for many organizations: The backlog of app project requests confronting every enterprise regardless of their internal app development capabilities Finding and employing better, faster tools to speed and simplify the process of developing those apps. The emergence of …

  • Live Event Date: September 23, 2015 @ 1:00 p.m. ET / 10:00 a.m. PT The cloud is not just about a runtime platform for your projects – now, you can do your development in the cloud, too. Check out this upcoming eseminar to learn how the cloud improves your development experience and team collaboration. Join Dana Singleterry, Principal Product Manager for Oracle Dev Tools, as he discusses how to simplify every aspect of the development lifecycle, including requirements gathering, version management, code …

Most Popular Programming Stories

More for Developers

RSS Feeds

Thanks for your registration, follow us on our social networks to keep up-to-date