High Performance Solution Ini File Class with MMF

WEBINAR: On-demand webcast

How to Boost Database Development Productivity on Linux, Docker, and Kubernetes with Microsoft SQL Server 2017 REGISTER >

Environment: VC6, WinNT/2k/XP (Having been tested in English/Japanese Win2k/XP)

Special Note: This Unicode Version is full-fledged, while the ANSI version does not support the section header map

Key Technology Used: Memory Map File (MMF), Win32 Memory Management API, Unicode, Data Structure

Applicable Article Category in CodeGuru: Miscellaneous Data Technique

Summary

All is from my previous project, which saves the data of the coordinate, text contents, color, font name, and size information of nearly 30000 dialog controls (listbox, static, edit, and so on) distributed on more than 100 windows. So, the total size of these controls will be at least 10*n Mega bytes and it's not suitable to use the Windows Registry. You see, using a native Ini file Win32 API is terribly impractical because the performance is too low. So, I designed and implemented the following Ini class using MMF; with the pre-allocated buffer and section header map technology, it should be the ultimate solution to the 5-50 Mega byte level Ini File reading and writing situations.

Implementation Description

Pre-allocated buffer

Please refer the following figure:



Click here for a larger image.

When I load the Ini File from the disk to the MMF, depending on the case, I allocate a bigger MMF than the original disk file. When the user just reads the Ini MMF, the Pre-Buffer (see the red bar part) will not be touched. Say, if the user writes more data into the Ini MMF, I can expand the Ini MMF into the Pre-Buffer part. Only when the Pre-Buffer is nearly used up (I call it a Critical Point, see the slim green line), I unload the whole MMF to flush the disk, and load it again with a larger MMF. When measuring carefully in your case, you can get a fairly good performance upgrade with this extra buffer.

Section head position map

When loading an Ini File from the disk file, I make a map of Section Head Pointer and the Section Name. This will take time, but later, when the user searches a section in the end part of the Ini MMF file, it will save considerable time. In my Ini File implementation, I used the CMap MFC class, so this class needs MFC library support. Because my previos project is an MFC-based application, it is reasonable, but to be more universal, I plan to implement it again later with STL.

Special Note with Unicode Ini File:

It is very important to know that Unicode text files have two formats in the Windows OS. And what makes things worse is that Microsoft changed its behavior when it issued Win2000 SP1. In the original Windows 2000, if you open an ANSI text file with Notepad and save it as Unicode text, it just calls the MultiByteToWideChar WinAPI to make changes and save the text; that's it. After you install Win2k SP1 and do the same thing, this time, Notepad adds heading 0xFF FE to the Unicode text. This lack of congruity belabored me a whole day because some member's machine in my team did not upgrade with Win2k SP. (BTW, some specification on the Internet says that in Mac machines the heading will be 0xFE FF.) My Ini File class will check the heading and cope with it.

Usage:

I just want to kiss it (keep it simple, stupid). Unlike some previous Ini File classes on this site, I use just two functions that have the same signature of the Win32 API—GetPrivateProfileString and WritePrivateProfileString to wrap my Ini File class. For example, instead of adding a RemoveSection function, you pass NULL to lpAppName in JXWritePrivateProfileString and you get it. When you want to get all key names in a given section, pass NULL to lpKeyName and get a double NULL ending buffer in lpReturnString. (I include a full-fledged demo project to show this.)

DWORD JXGetPrivateProfileString(
  LPCTSTR lpAppName,          // section name
  LPCTSTR lpKeyName,          // key name
  LPCTSTR lpDefault,          // default string
  LPTSTR lpReturnedString,    // destination buffer
  DWORD nSize,                // size of destination buffer
  LPCTSTR lpFileName          // initialization file name
) ; 

BOOL JXWritePrivateProfileString(
LPCTSTR lpAppName,            // section name
LPCTSTR lpKeyName,            // key name
LPCTSTR lpString,             // string to add
LPCTSTR lpFileName            // initialization file
);

Both support ANSI and Unicode versions. Just include JXIniFile.h in your file and add both the JXIniFile.h and .cpp to your project. Check MSDN to get the detailed information because they are the same signature as the Win32 API.

There's just one thing I decided to be different from the Win32 API. When you want to get all the section or key names, and you pass a buffer to JXGetPrivateProfileString, a return value of 0 means your buffer is too small, while the Win32 API will return the buffer size needed. So, when you get the return of 0, just reallocate your buffer and try it until you pass. I did this because the program usually knows the rough size it takes to get the list of section/key names, and a mid-way return 0 in my Ini File class saves time.

Another thing is that I permit the user to unload the Ini MMF at any time by calling

BOOL JXUnloadPrivateProfile();

One possible use of it is that when you debug the program, you can unload the MMF, use a text editor to change some value, and ask your program read the Ini disk file again to see the effect.

Demo

I provide a simple, dialog-based program to show the usage of these two functions. The code to enumerate sections and keys is used to read a Ini file; usually, the program has knowledge of the section name and key name. I enumerate all the section and key names before fetching the value, just to give a worst-case example.

You press the buttons from left to right, generating 100 (by default) sections, each including 100 keys, the value is randomly generated; then, writing the data to disk, reading it back to the right side tree to confirm. The popped-up time message box tells you the total time to read the Ini File and fill the tree. (You can comment the tree-filling code and see the time of reading exclusively.)

Are You Ready To Use It?

  1. If your application does not use MFC, you have to link with the MFC library because I use CMap to implement the section head map. I am considering implementing it again with STL to make it more universal.
  2. If your application is not Unicode based, by default you will use its ANSI version. In that case, no section header map support is available. (Just because I have no time and need to use it.) It means the performance is not as good as its Unicode counterpart, but still better than the Win32 API (at least >> 10 times faster). Besides, you can still explicitly call JXGetPrivateProfileStringW and JXWritePrivateProfileStringW.
  3. Consider your data structure. Instead of using 10 sections each including 1000 values, 100 sections with 100 values will be better.
  4. Decide your pre-buffer size. I use 1M by default; maybe it is overkill in your case. At least it should keep MMF from reloading in a few seconds in most cases. Actually, if the program just reads the Ini data, you can use 4K (1 memory page).

Acknowledgements

I would like to express my thanks to the following persons and institute: Prof. Moriya Shinji, Human Interaction Lab of Tokyo Denki University, Media Center of Tokyo Denki University, for their academic direction, warm-hearted help, and continuous encouragements to my career life.

Also thanks go to FujiSoftware ABC, Ltd., for the developer position offered, plus their wage.

Downloads

Download demo project - 54 Kb (JXIniFile source file includes <about 4000 lines of code to deal characters one by one>, + exe)
Download Demo Exe File Only - 44 Kb (Only exe demo, MFC dynamic linked)

Version History

Version Release Date Features
1.0+ Nov. 5, 2002 (Add a demo, finish this article)
1.0 Sept. 27, 2001 Heading 0xFF FE handling


Comments

  • How about hierarchical structure of saved info?

    Posted by Legacy on 11/20/2002 12:00am

    Originally posted by: vgrigor

    Automatical Extensibility compatibility is right feature and need
    of update programs,
    that is advice of "named" ini strings over old MFC "just effective" serialize() (not stable for possible errors also), now not very need over better structure(! -that's it.) and reliability(!) -(how about it in your code?),(not makes you kicked 'after')
    but if you want to save hierarchical structure into file
    what is the solution except XML?
    - features need To make it universal saving solution.

    how about versioning in your solution?

    Reply
  • XML is better

    Posted by Legacy on 11/17/2002 12:00am

    Originally posted by: M.

    XML is better to use for saving configuration to a file. It has the same structure as the registry and it's much more faster than plain .ini file.

    Reply
  • Use a very large MMF

    Posted by Legacy on 11/15/2002 12:00am

    Originally posted by: Neville Franks

    Hi Zhefu,
    It is my understanding that you can allocate a very large MMF without it taking up any more disk space of memory than is actually being used at any point in time. Simple tests I've done indicate this is correct. This means you can say allocate a 1G MMF and not worry about having to resize it.

    You can also resize an MMF on the fly, however depending on the version of Windows you are using and possibly other things the base address can and will change. If the MMF stores pointers then you would need to adjust them if this resizing technique is used.

    Neville Franks. www.getsoft.com

    Reply
  • About 0xFEFF(A BOM of Unicode)

    Posted by Legacy on 11/15/2002 12:00am

    Originally posted by: chenrs

    The character 0xFEFF is the Byte Order Mask(BOM) of Unicode which serves to indicate the endianness of a sequence of bytes,its byte-swapped BOM(BSBOM) 0xFFFE is defined to never be a valid Unicode character.

    When you convert some Unicode characters in memory into a sequence of bytes for storage (or transmission),it is a good idea to put an BOM(0xFEFF) as the first two bytes of the file.

    //want to save the text "ABC" to the file "fd"
    WCHAR wcTEXT = _T("XABC");
    wcTEXT[0] = 0xFEFF; //NOT 0xFFFE
    fwrite(wcTEXT,sizeof(WCHAR)*4,1,fd);

    In the little endian(LE) machine architectures(such as Intel x86 CPU),it will be saved as:
    "FF FE 41 00 42 00 43 00"
    But in the big endian(BE) platforms(Sun Solaris,MAC PowerPC CPU),it will be saved as:
    "FE FF 00 41 00 42 00 43"
    So you can check the first two bytes of an unicode(UTF-16) file to establish its endianness.

    You can use your NotePad open a text file,save it as "Unicode"(LE) and "Unicode Big endian",you will find the difference by comparing them with UltraEdit or other HexEditor.

    More information will be available on the IBM web site:
    http://www.ibm.com/developerworks/unicode/library/utfencodingforms/index.html?dwzone=unicode

    Reply
Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • As all sorts of data becomes available for storage, analysis and retrieval - so called 'Big Data' - there are potentially huge benefits, but equally huge challenges...
  • The agile organization needs knowledge to act on, quickly and effectively. Though many organizations are clamouring for "Big Data", not nearly as many know what to do with it...
  • Cloud-based integration solutions can be confusing. Adding to the confusion are the multiple ways IT departments can deliver such integration...

Most Popular Programming Stories

More for Developers

RSS Feeds

Thanks for your registration, follow us on our social networks to keep up-to-date