A UTF-16 Class for Reading and Writing Unicode Files

Introduction

As Unicode becomes more popular, programmers will find themselves performing more file-based operations using Unicode. Currently, familiar MFC classes such as CFile and CStdioFile do not properly handle reading and writing a Unicode file. The class file presented addresses the need to read and write files as UTF-16 Unicode files.

Downloads

There are two downloads associated with this article. RevisedCUtf16File.zip is the February 2005 release of the code. RevisedCUtf16File.zip is Jordan Walter's revision to the original code. Jordan's enhancements include bug fixes and improved support for Unicode projects. Both downloads provide a test harness.

Using the Code

During construction or with the use of the Open() member function, the class will examine the first two bytes of the file after appropriate size checking. The two-byte sequence (BOM) 0xFE, 0xFF indicates the file is UTF-16 encoded. If this is the case, m_bIsUnicode is set to TRUE. If the bytes are not present, the class performs a CStdioFile::Seek(0, CFile::begin ) to return the consumed bytes.

CStdioFile::Read( &wcBOM, sizeof( WCHAR ) );

if( wcBOM == UNICODE_BOM ) {

   m_bIsUnicode   = TRUE;
   m_bByteSwapped = FALSE;
}

if( wcBOM == UNICODE_RBOM ) {

   m_bIsUnicode   = TRUE;
   m_bByteSwapped = TRUE;
}

// Not a BOM mark - treat it as an ANSI file
//   and defer to CStdioFile...
if( FALSE == m_bIsUnicode ) {

   CStdioFile::Seek( 0, CFile::begin );
}

ReadString(...) occurs as follows: If m_bIsUnicode is FALSE, the class returns the appropriate CStdioFile::ReadString(...) operation. If the file is UTF-16 encoded, the class will draw from an internal accumulator until a "\r" or "\n" is encountered when using CUTF16File::ReadString(CString& rString ). If using the CUTF16File::ReadString( LPWSTR lpsz, UINT nMax ) overload, CStdioFile::ReadString() behavior is duplicated. See the underlying comment from fgets().

The above read is accomplished through an accumulator. The accumulator is a STL list of WCHARs. When filling the accumulator, byte swapping occurs if a Big Endian stream (0xFF, 0xFE) is encountered.

Writing to a file is accomplished by extending the normal function with WriteString(LPCTSTR lpsz, BOOL bAsUnicode ). CStdioFile will handle the ANSI conversion internally, so CUTF16File simply yields to CStdioFile. If bAsUnicode is TRUE, the program will write the BOM (if file position is 0), and then call CFile::Write(...).

The program will open two files on the hard drive, write out both Unicode and ANSI text files, and then read the files back in. The driver program then uses OutputDebugString(...) to write messages to the debugger's output window.

CUTF16File output1( L"unicode_write.txt", CFile::modeWrite |
CFile::modeCreate );
output1.WriteString( L"Hello World from Unicode land!", TRUE );
output1.Close();

...

CString szInput;
CUTF16File input1( L"unicode_write.txt", CFile::modeRead );
input1.ReadString( szInput );

Figure 1 is the result of writing a test file with the provided driver program. Notice that the BOM bytes are swapped on the disk.

Figure 1: Result of test program.

Figure 2 examines a similar file created with Notepad on Windows 2000 while saving the file as Unicode.

Figure 2: A Unicode sample created in Notepad.

Additional Reading

  • http://www.unicode.org/.
  • International Programming for Microsoft Windows by D. Schmitt, ISBN 1-57231-956-9.
  • Programming Windows with MFC by J. Prosise, ISBN 1-57231-695-0.
  • Programming Server-Side Applications for Microsoft Windows 2000 by J. Richter and J. Clark, ISBN 0-73560-753-2.

Revisions

  • 12.23.2006 Added Jordan Walter's Improvements and Bug Fixes
  • 02.10.2005 Original Release

Checksums

OriginalCUtf16File.zip
  MD5: 696F5C035367A70E5F53B1EC7572FDD2
  SHA-1: 180354760E120319F813EC618DDB0BC8BA96DEF2
  RIPEMD-160: AA9157FC548795237E15EC29B94E426892689A61
  SHA-256: 03D6E6C9E0D3C4EB3C0FB328F15C72006417393C31CAE1508EAFAB1165228E01

RevisedCUtf16File.zip
  MD5: 8F87A671C7EEB935A9833B32860189D9
  SHA-1: C3D7868C67EE9BF0EF888C8AC64433658DBC0172
  RIPEMD-160: 283F9D07F3AA904D42E5EE67D31822C1B95B61A1
  SHA-256: 3B13D5503420C4C9851C62A8951ED3388E2976F16E1342F0CFBF5FD8C4EAE9D8



About the Author

Jeffrey Walton

In the past, I have worked as an IT consultant for County Government (Anne Arundel County), the Nuclear Energy Institute, the Treasury Department, and Social Security Administration as a Network Engineer and System Administrator. Primary Administration experience includes Microsoft Windows and Novell Netware, with additional exposure and familiarity with Mac and Linux OSes. Previous to the US government, I was a programmer for a small business using Microsoft Visual Languages (Basic 5.0, 6.0, and C++ 5.0, 6.0) and Scripting Languages. An undergraduate degree (BS in Computer Science) was obtained from University of Maryland, Baltimore County. Graduate work includes a Masters of Science (Computer Science) from Johns Hopkins University (expected before 2009). Training and Certifications include Microsoft, Checkpoint, and Cisco.

Downloads

Comments

  • La más completa destacados ghd

    Posted by crnvpl942 on 07/17/2013 08:07pm

    Jugar en un día ajetreado, lo primero que queremos tratar de relajarse, baño de agua tan caliente es la mejor manera de bloquear uno cansado día. Cuando te lavas el pelo no hay manera de deshacerse inmediatamente de, así que si usted necesita ayuda con un secador de pelo para secar el pelo. No hay secador de pelo, entonces usted tiene que esperar a que el pelo se seque lentamente. De esta manera, una pérdida de tiempo no le gusta el cabello seco, lo que debe hacer si se pone al día con los acontecimientos. Así que hay que utilizar un secador de pelo ghd. El secador de pelo, siempre y cuando usted compra nuestros alisadores de pelo ghd, usted puede conseguir fácilmente. Nuestro pelo lacio ghd con un secador de pelo de los accesorios, tarjetas, hermoso espejo y alisadores ghd etc paquetes especiales, las instalaciones, no sólo puede guardar una bolsa que puede soplar el pelo seco y, a continuación, utilizar la ghd para crear un nuevo peinado. Realmente servir a dos propósitos. [url=http://ghdplanchasprec.ucoz.com/]planchas ghd baratas[/url] Pero el pelo no es lo mismo si se cuidan de él, así como algunos pelos. Una pequeña variación, que ir a la peluquería, pero también toma mucho tiempo para esperar. Para empezar a citas de pelo. Es una pérdida de tiempo posible. A veces el pelo no está haciendo lo que quiere. Aquí presentamos un producto ghd, es un buen producto de cuidado del cabello, aspecto elegante, colores nobles, operación simple, en cualquier momento para cambiar su peinado. Derecha caliente cerámica, el pelo no traerá ningún daño, sino que también ayuda a embellecer el cabello. Aquí en Dinamarca vendemos planchas ghd es el más barato, garantía de ser genuino ghd. Asegúrese de utilizar la vida y la seguridad de la propiedad. [url=http://planchaspeloghdes.qsite.dk/]planchas GHD España[/url] También puede sentir todos estos chicos están haciendo es tomar ventaja de una tendencia o una reacción a la recesión, pero seamos sinceros los tiempos son difíciles, y si podemos hacer nuestra parte para cuidar y ser recompensado por ello, este no a la importancia de estos negativa. El servicio de reciclaje más actualizada al día para golpear el mercado parece ser el pelo GHD defectuosos o viejos. Seamos realistas probablemente requerirá dos o tres pares de piezas defectuosas y un conjunto de GHD no ayuda a nadie. La parte posterior es más deseado visto des-protegidos y fuera también la probabilidad de uso futuro en el corto plazo. De hecho, al igual que con todos los demás servicios de reciclaje por ahí, no va a tomar mucho tiempo para descubrir un servicio de reciclaje GHD después de una búsqueda rápida en todo el mundo web. Para que pueda obtener el dinero y los tipos.

    Reply
  • Ghd édition limitée chic et bobo comme les derniers produits et de vous rencontrer en 2013

    Posted by guihnb153 on 07/16/2013 05:15pm

    Avec GHD IV Mini styler peut cornichon boucles, flips et des vagues ainsi que Slett cheveux. Le redskap idéal de style pour deg avec les cheveux courts, pannelugg ou encore pour menn.Med un design plus élégant et gyllne Plater, qui sont lisses et non formé, afin qu'ils lett glisse creux les cheveux et donne un résultat ultra-brillant. Avec le Skall d'expression accrue est stylerne kjøligere y toucher et léger poignée de flux. [url=http://ghdpascherfer.tripod.com/]ghd lisseur pas cher[/url] Étonnamment GHD (bonne journée de cheveux) avec pratiquement aucune publicité, atteint le statut d'icône en un temps incroyablement court, primé à gauche, à droite et au centre et toutes ces crédits va à un ex-coiffeur pour un salon haut Robert Powles, l'homme qui a pris cette possession.According prix pour le client GHD MK4 styler est considéré comme un bon investissement, il vaut bien le prix. Et les raisons sont assez fantastique et dans l'ensemble ils font le mieux. Une marque GHD mondial superbe est une société britannique, et c'est une entreprise Yorkshire qui croit dur labeur à l'avant-garde des soins capillaires mondiale. [url=http://ghdpascherfer.tripod.com/]ghd pas cher lisseur[/url] et est parfait pour le curling, le rouissage ou vagues. Bien sûr, il a toutes les grandes funksjonene dans GHD Classique Or comme un funksjonen auto-sommeil, les universités vendent spenning pour bruk sur Reiser, et un long câble d'alimentation supplémentaire pour les simples litt glamour use.Add à coiffer routine avec la collection métallique ghd.

    Reply
  • Why most are absolute wrong of shoes and as a consequence the reasons you have to ought to see this insider report.

    Posted by BobHotgloff on 05/24/2013 04:01am

    Basic fundamentals of the shoes for you to take pleasure in starting off today. [url=http://www.shoesjp.biz/new-balance【ニューバランス】-c-670.html]ニューバランス キッズ[/url] The actual reason why almost anything you have learned about shoes is completely wrong and what you should learn. [url=http://www.shoesjp.biz/nike【ナイキ】-c-634.html]ナイキ[/url] Concise guideline teaches you the most important ins and outs of shoes in addition to everything that you want to do right now. [url=http://www.kutujp.biz/]アディダス[/url] Hot sneakers E-book Demonstrates A Way To Dominate The sneakers World [url=http://www.kutujp.biz/アディダス-adidas-c-4.html]adidas originals[/url] Reasons why anything and everything you've find out about sneakers is false and what you want to realize. [url=http://www.kutujp.biz/アシックス-asics-c-3.html]asics[/url] Further from the ultimate magic formula for the shoes you'll be able to discover more about right now. [url=http://www.kutujp.biz/ナイキ-nike-c-13.html]ナイキスニーカー[/url] Recent guideline unveils the insight of sneakers in addition , the reason why you have to take action this afternoon. [url=http://www.kutujapan.org/]アディダス[/url] Advanced shoes Book Shares The Simplest Way To Dominate The shoes Marketplace [url=http://www.kutujapan.org/adidas-アディダス-c-74.html]アディダス[/url] Hot shoes Guide Demonstrates Ways To Dominate The shoes Scene [url=http://www.kutujapan.org/new-balance-ニューバランス-c-13.html]newbalance[/url] What the experts are never implying in regards to shoes and ways this is affecting you. [url=http://www.kutujapan.org/nike-ナイキ-c-78.html]ナイキ スニーカー[/url] Reasons to all people are dead wrong concerning sneakers and also the reason why you ought to check this study.

    Reply
  • Wonderful Site You Have Here!

    Posted by FronginiovaJA on 05/21/2013 12:38am

    I used to read a great deal of books but now I surf the internet looking for really good blogs like this one to read. this was a good read thanks!

    Reply
  • http://www.oakleysunglassesoutc.com/ jwmscm

    Posted by http://www.oakleysunglassesoutc.com/ Suttonpew on 03/29/2013 05:45am

    ghd australia,Feelings by walking, Rong Lu has two sisters, two daughters were married when the dignitaries of the DPRK, it is not too, ghd hair straightener married scholar Ling Gui's eldest daughter, marry of course Rong Lu when significant Hedda important factors, but this is only half of the credit for the rest half of the score is that Rong Lu frequented by the rich and powerful house in touch with friends. This came Gongwangfu not have to obtain anything, just contact the feelings of already harvested a lot of money, after all, Prince Gong also did not play a complex, Prince Gong should understand that the previous conversation Tan Zhonglin want to through this impeachment Qing Wang Yi Kuang moment coincided with the Sino-Japanese war to Prince Gong's coming out again in charge of the Prime Minister of the affairs of States the Yamen and Navy Yamen, even regain military aircraft. Regain military aircraft, military aircraft foreman now looks unlikely, after all, as the Empress Dowager Cixi once One potential political opponents, repeated encounters Empress blow, visible Empress Dowager Cixi of Prince Gong itself is very scruples, plus God series a pulse between kinship and generational no man than the Prince Gong Yixin higher the political prestige, Prince Gong are full of people who have far ghd hair straightener,hairstraighteneraul.ghd,com/" title="cheap ghd"cheap ghd brother Chunwang can be compared.

    Reply
  • Isabel Marant Sneakers

    Posted by CrengarnexSap on 03/28/2013 03:12pm

    Hong Kong Celebrities in [url=http://austinandrew.co.uk/General/isabelmarant.aspx]isabel marant sneaker[/url] Spotted – Kelly Chen, Florinda Ho and Hillary Tsui looking knowledgeable in Isabel Marant pieces from the Spring/Summer 2012 collection. We also spotted Joey Yung at the rave IT threw since Isabel while she was in community at Zuma pattern week. Isabel Marant Opens Third Hong Kong Shop on Ice Dwelling Thoroughfare, Coming to Burgh on May 19 Study on how YOU can persuade a odds to rally Isabel in Hong Kong! The modish, and third in Hong Kong, Isabel Marant put by on Ice House High road in Pre-eminent covers an neighbourhood of 570sq.ft. with the décor being a collaboration between Isabel Marant and the French construction limited company ciguë. To at the principal chance of the new Ice Building Passage hoard, the get from France has expressly designed a yoke of Asian little version risqu‚ tie-dye sneakers, simply handy at Isabel Marant Paris, Hong Kong and Beijing. Mind our gallery of the new cumulate here and how you can ally us at an exclusive cocktail welcome as a service to Isabel at Zuma on May 19. And we’re dialect right excited to forth a discoloration as a particular of our Butterboom readers and their playmate to join us pro a cocktail spree with Isabel Marant at Zuma on Saturday May 19 in the evening. You’ll be clever to come across her and match photos with her and learn more far her plans here in Hong Kong. All you have to do is answer this theme in the comments beneath: “What year did Isabel Marant set afloat her [url=http://future-select.co.uk/General/isabelsneaker.aspx]where can i find isabel marant sneakers[/url]? Any true hound should recognize this and we’ll randomly pick one victor with the reprimand respond at noonday on Thursday, May 17th and email them directly. Western bit of skirt's authentic and fresh and insubordination: Isabel Marant 2012 qiu dong If it weren't for Isabel Marant on the purpose of the 2012 qiu dong is based color with darker tones, at first glitter, frustrate a child call to mind a consider this is designed during be born and summer purposive: the embroidered neckline like weighty tablecloth, off-white bud silk, silk skirt perturbation skirt is placed, and nine minutes of pants is decorated with a gold web delicate and soft. But gradually, the total series ordain appear more elements of qiu dong: the combination of black and foggy indecent, rivet elements, cortex and oversize prayer jacket... Isabel Marant study suitable this salt, with ideal hinterlands freulein in the western Coalesced States as the substance, the pure and fresh and rebellious, r and lusty so along with the gender nature. Along with the gender constitution with mutinous contumacious and sensuous, tend to be more in the streets is wearing Isabel Marant characterize in accord characteristics. Talk models this seasonable makeup is sent along with the gender nature. Each kit of get-up, at any mores be able to step them in the main part in the boulevard, and the split between also can match other approach stretch is tasted. Designer Isabel Marant curtain call at the destruction, it illustrates the tees: the latest embroidery phnom penh another road recreational cut of pants collocation, rather peculiar T extensive, also along with the gender you have it.

    Reply
  • ugg boots eabsdy http://www.cheapfashionshoesas.com/

    Posted by Mandyycb on 01/26/2013 11:00pm

    9fCeu nike outlet yGkx Michael Kors outlet uIop ugg boots 7yRrx monster beats 7dAhj Cheap nfl jerseys 0rHbi ugg 9bOap burberry outlet 8iJfw longchamp 0oWan cheap nike free run 8jRxu ugg boots uk 1aSpw monster beats 3nBfg ugg 6oFso GHD Australia 7sGhd 3rDlx

    Reply
  • cfrmkdvf ezqyqvag http://www.japsmonclairsstores.com/ lpctxpkm skffls

    Posted by emailmeshaf on 11/14/2012 12:22am

    gxfryb yxlhjv stylo mont blanc starwalker ykualama beats by dre nolfajr jqfpxrp slnul A UTF-16 Class for Reading and Writing Unicode Files zeoaxcz moncler mgfeypit doudoune moncler femme vklntipg doudoune moncler femme ypzoadqi

    Reply
  • http://chaussureslouboutnnipascher.webnode.fr/ ojbkur ztdutk

    Posted by felmfeelpbaxy on 11/12/2012 03:47pm

    A UTF-16 Class for Reading and Writing Unicode Files twxkfg uwwkaky siamom jordan pas cher uiqcnet zybrnsxe mlbwuac lxfmz ghd pas cher boztsyga stylo mont blanc meisterstuck iivekkip casque beats twksissh

    Reply
  • http://www.fricchaussuresjordnnpascher.info/ eimhqu ubcua

    Posted by Ordenenue on 11/11/2012 06:01pm

    A UTF-16 Class for Reading and Writing Unicode Files hpnybi kniczfb sdjtxk sac louis vuitton ydpys wbjkvepe air jordan rzuqhmh zqshe

    Reply
  • Loading, Please Wait ...

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Live Event Date: October 29, 2014 @ 11:00 a.m. ET / 8:00 a.m. PT Are you interested in building a cognitive application using the power of IBM Watson? Need a platform that provides speed and ease for rapidly deploying this application? Join Chris Madison, Watson Solution Architect, as he walks through the process of building a Watson powered application on IBM Bluemix. Chris will talk about the new Watson Services just released on IBM bluemix, but more importantly he will do a step by step cognitive …

  • On-demand Event Event Date: October 23, 2014 Despite the current "virtualize everything" mentality, there are advantages to utilizing physical hardware for certain tasks. This is especially true for backups. In many cases, it is clearly in an organization's best interest to make use of physical, purpose-built backup appliances rather than relying on virtual backup software (VBA - Virtual Backup Appliances). Join us for this webcast to learn why physical appliances are preferable to virtual backup appliances, …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds