A Simple CString Tokenizer

WEBINAR: On-demand webcast

How to Boost Database Development Productivity on Linux, Docker, and Kubernetes with Microsoft SQL Server 2017 REGISTER >

As the name suggests, this is a simple class to extract tokens from a CSting. I wrote this class becuase during the course of my final year project at University I needed a simple way to extract 'tokens' from a CString. This is what I came up with. Its probably not the best or most effeicent way to accomplish the task - but it works which in the main thing.

To use the CToken class:

-First include the CToken header file in your porgram:

#include "Token.h"

-Then you are free to create and use an instance of CToken. For example:

CString str = "A B C D"
CString newTok;
CToken tok(str);
tok.SetToken(" ");
while(tok.MoreTokens())
{
   newTok = tok.GetToken()
}
Improvements:
If people can suggest any improvements or there bugs please drop me an email

Download demo project - 15 KB

Download source - 3 KB



Comments

  • help

    Posted by umar on 01/25/2013 02:11am

    hi, I am a beginner in C++. And i am very thankful to you if answer my question kindly. i want to know that if i save a file with a lot of data using ofstream then if i want to find a specific data from whole file. who can i do it.How can i make a struct or function for it. what is the coding please tell me.

    Reply
  • problem: blank tokens are trimmed!

    Posted by pamar on 11/15/2007 11:35am

    Whit this string: "A|B|C| " and | separator, last token must be " " but it is trimmed.

    Reply
  • i rewrote it : shorter & better

    Posted by Legacy on 03/10/2003 12:00am

    Originally posted by: joeri

    just for those in need, i thought this delimiter wasn't working well, so i rewrote it:

    class CToken
    {
    public:
    virtual CString GetNextToken(); // Returns the next token
    virtual BOOL MoreTokens(); // Do we have any more tokens to retrieve
    int m_nAmount;
    int m_nCurrent;
    CToken(CString str,CString token=" ");
    virtual ~CToken();

    protected:
    CString m_strToToken; // The original string to tokenize
    CString m_strLeft; // The string we have left after taking tokens out
    CString m_strToken; // What is the value to tokenize on?
    private:
    int GetAmount(CString str);
    };

    CToken::CToken(CString str, CString token)
    {
    str.TrimLeft();
    str.TrimRight();
    m_strToToken = str;
    m_strLeft = str;
    m_strToken = token;
    m_nAmount = GetAmount(str);
    m_nCurrent = 0;
    }

    CToken::~CToken()
    {

    }

    int CToken::GetAmount(CString str)
    {
    int count = 0;

    int pos, len;

    len = str.GetLength();
    pos = str.Find(m_strToken,0);
    while (pos != -1)
    {
    count++;
    pos++;
    str = str.Right(len - pos);
    str.TrimLeft();
    len = str.GetLength();
    pos = str.Find(m_strToken,0);
    }
    if (!str.IsEmpty()) count++;
    return count;
    }
    BOOL CToken::MoreTokens()
    {
    if (m_nCurrent < m_nAmount)
    return TRUE;
    else return FALSE;
    }


    CString CToken::GetNextToken()
    {

    if(!MoreTokens())
    return "";
    CString ret;
    int pos, len;

    len = m_strLeft.GetLength();
    pos = m_strLeft.Find(m_strToken,0);
    if (pos != -1)
    {
    ret = m_strLeft.Left(pos);
    pos++;
    m_strLeft = m_strLeft.Right(len - pos);
    m_strLeft.TrimLeft();
    }
    else ret = m_strLeft;
    m_nCurrent++;
    return ret;}

    Reply
  • lexiacal analyzing

    Posted by Legacy on 01/12/2003 12:00am

    Originally posted by: alan

    helloo  friend,
    
    i want a help from u?now i am doing a project with some lexiacal analysing data.actually my problem is i have to do this with a group of data.from the given passage or text i have to find out the stake hilders needs,alterables and constraints.i have to identify these thibgs from the text.for this i have to search each and every string.could u please suggest some tools to identify these things from a passage.if could it will a great help for me.and i am doing this with c++ and flex (unix programming toool).hindly plzz suggest some logics and better sites from there i could find sample codes....... thanking you
    alan

    Reply
  • Nice class

    Posted by Legacy on 07/31/2002 12:00am

    Originally posted by: Annette

    Hi,

    I just wanted to say that even if it a simple class, I thought it was very nice and helpful. I like working with CString instead of "hacking" along with all these char * and other string types.

    So, thanks!!

    Annette Skaar

    Reply
  • pointer to a pointer in a string class

    Posted by Legacy on 05/06/2001 12:00am

    Originally posted by: Cris Moore

    I'm working on a project. My data member in the class is char **list. The parameter I'm being passed is char *alist. How do I convert?

    Reply
  • I need help!!

    Posted by Legacy on 04/18/2001 12:00am

    Originally posted by: John

    I am a beginner in visual C++ programming.I have to do a program that read a multi-command file . Each command have a fixed number of parameters .I have to make a string analysis and based on that analysis the program should fill some small records like matrix (two dimension vectors).
    What would be the best way to do this?(in special the string analysis).


    Reply
  • I wouldn't bother using this class....

    Posted by Legacy on 09/06/2000 12:00am

    Originally posted by: Paul Wendt

    Hey,
    This class doesn't really offer anything above strtok so I'd personally stick with strtok. It's not like there is a huge learning curve with learning strtok either :)
    --paul wendt

    Reply
  • More easy way is to use AfxExtractSubString() function :))

    Posted by Legacy on 02/11/2000 12:00am

    Originally posted by: German Madyarov

    See afxwin.h and you will find this code:
    
    

    BOOL AFXAPI AfxExtractSubString(CString& rString, LPCTSTR lpszFullString,int iSubString, TCHAR chSep = '\n');

    Reply
  • one single token not recognized?

    Posted by Legacy on 11/09/1999 12:00am

    Originally posted by: Hasch

    When I have only one token in the string, say "B" without any delimiters, I would expect to get "B" on calling GetNextToken. But MoreTokens() returns "false".
    I think, the "Check to see if there anymore tokens" from GetNextToken should also be done in the constructor. Then the flag "m_bFinalToken" would be set correctly.
    What do you think of it?

    Reply
  • Loading, Please Wait ...

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • As all sorts of data becomes available for storage, analysis and retrieval - so called 'Big Data' - there are potentially huge benefits, but equally huge challenges...
  • The agile organization needs knowledge to act on, quickly and effectively. Though many organizations are clamouring for "Big Data", not nearly as many know what to do with it...
  • Cloud-based integration solutions can be confusing. Adding to the confusion are the multiple ways IT departments can deliver such integration...

Most Popular Programming Stories

More for Developers

RSS Feeds

Thanks for your registration, follow us on our social networks to keep up-to-date