CTokenEx – A String Tokenizer Class

Environment: VC++ 6.0 SP3 (using Warning Level 4 in RELEASE Mode), Environments Tested: WinNT 4.0 SP4, Win95

Basically, I saw other string tokenizers here (CodeGuru) and they lacked the functionality I was
looking for.  Therefore, I created one for myself using the KISS method.  This is a
VERY SIMPLE sample!!!!

Here is a summary of the functionality in the CTokenEx class:

  • Can use “SplitPath” to break-up the path into sections (Drive/Sharename, Directory,
    Filename, Extention).  Also, recognizes UNC Names (which _tsplitpath doesn’t).

  • Can use “Join” to create a CString from a CStringArray with delimiters of your choice.
  • Can use “Split” to break-up a CString into a CStringArray (according to the delimiter).
  • Can use “GetString” to get the first sub-string in a CString (according to the delimiter).


The “Split” and “GetString” functions recognize multiple delimiters as an empty string so that
it will NOT add blanks to an Array.  See example code below: :

Say you have a CString that contains: “abc,def,,,ghi,,jkl,,”

// Split Function
// Split will fill an array with:
// String Position
// ====== ========
// abc 0
// def 1
// ghi 2
// jkl 3
void CTokenDlgDlg::OnSplit()
CTokenEx tok;

// CString for the Split Function
CString csSplit = “abc,def,,,ghi,,jkl,,”;

// CStringArray to fill
CStringArray SplitIt;

// Call Split
tok.Split(csSplit, “,”, SplitIt);

// GetString Function
// GetString will return a string:
// abc
// …and more calls to GetString will return a strings:
// def
// ghi
// jkl
void CTokenDlgDlg::OnGetstring()
CTokenEx tok;
char Buf[254];

CString csRef = “abc,def,,,ghi,,jkl,,”;
do {
CString csRet = tok.GetString(csRef, “,”);
// Do something with the returned value.
} while (!csRef.IsEmpty());

I hope that others find this class useful.


Download demo project – 37 Kb
Download source – 99 Kb

More by Author

Must Read