An alternative Regular Expression Class

This is another regular expression library. Like Zafir's it is based upon the work of Henry Spencer. I started using this a long time ago and called my class Regexp (rather than CRegExp). Actually I prefer Zafir's name but I have too much code using the other name to want to change it, so right now my class is called Regexp (change it if you like).

So why put up another version? I hear you ask. Well the two classes took the same base code and then developed to solve different problems. CRegExp is geared to Search and Replace operations whereas Regexp was written to simplify tokenisation. I wanted a class that could be given a 'program' and from that, return specific substrings from it's input. Regular expressions may not be the fastest way to parse input (though with careful anchoring they can be made so that they fail quickly if they are going to) but once you have a working library they do allow for fairly rapid coding. On the whole this is good enough, worry about making it faster once you have it working and actually know that your optimization effort isn't going unnoticed.

For example:

Regexp re( "^[\t ]*(.*)[\t ]*\\((.*)\\)" );
CString str( "wyrdrune.com!kelly (Kelly)\n" );
CString name, addr;

if ( re.Match( str ) && re.SubStrings() == 2 )
{
	name = re[2];
	addr = re[1];
}

Will give:

name == "Kelly" and addr == "wyrdrune.com!kelly"

If you decompose the regular expression you get:

BTW: the phrase tagged regular expression refers to any part of the regular expression that is, because it was surrounded by parenthesis, accessible after a match has been made as a separate substring.  See here for more information about Regular Expression syntax.

In English, we are looking for two fields. The first will be all characters from the start of the line through to the second field (without any surrounding white space), and the second will be all characters within parenthesis following the first field.

The Class

The library itself comes as two source files, Regexp.cpp and Regexp.h. The header defines the Regexp class with the following members:

Regexp::NSUBEXP

A constant defining how many subexpressions that the library will support (usually 10), attempting to use a regular expression with more than this number will generate an error.

Regexp::Regexp()

A boring constructor, this must be initialized by assignment before anything useful can be done with it.

Regexp::Regexp( TCHAR * exp, BOOL iCase = 0 )

exp :

The regular expression itself, this format of which is defined later. The success or failure of the compilation can be discovered by using either GetErrorString() or CompiledOK().

iCase:

If TRUE the regular expression is compiled so that differences in case are ignored when matching.

Regexp::Regexp( const Regexp &r )

Construct a new regular expression taking the compiled form from another Regexp.

const Regexp::Regexp & operator=( const Regexp & r );

Assign Regexp r to the current object.

bool Regexp::Match( const TCHAR * s );

Examine the TCHAR array s with this regular expression, returning true if there is a match. This match updates the state of this Regexp object so that the substrings of the match can be obtained. The 0th substring is the substring of string that matched the whole regular expression. The others are those substrings that matched parenthesized expressions within the regular expression, with parenthesized expressions numbered in left-to-right order of their opening parentheses. If a parenthesized expression does not participate in the match at all, its length is 0. It is an error if this Regexp has not been successfully initialized.

int Regexp::SubStrings() const;

Return the number of substrings found after a successful Match().

const CString Regexp::operator[]( unsigned int i ) const;

Return the ith matched substring after a successful Match().

int Regexp::SubStart( unsigned int i ) const;

Return the starting offset of the ith matched substring from the beginning of the TCHAR array used in Match().

int Regexp::SubLength( unsigned int i ) const;

Return the length of the ith matched substring

Using the same example Regexp as before:

Regexp re( "^[\t ]*(.*)[\t ]*\\((.*)\\)" );
CString str( "wyrdrune.com!kelly (Kelly)\n" );
if ( re.Match( str ) && re.SubStrings() == 2 )
{
	ASSERT( re.SubStart(0) == 0 );
	ASSERT( re.SubLength(0) == 26 );

	ASSERT( re.SubStart(1) == 0 );
	ASSERT( re.SubLength(1) == 19 );

	ASSERT( re.SubStart(2) == 20 );
	ASSERT( re.SubLength(2) == 5 );
}

CString Regexp::GetReplaceString( LPCTSTR source ) const;

After a successful Match you can retrieve a replacement string as an alternative to building up the various substrings by hand.

Each character in the source string will be copied to the return value except for the following special characters:

&	The complete matched string (sub-string 0).
\1	Sub-string 1
...	and so on until...
\9	Sub-string 9

So, taking the now ubiquitous example:

CString repl = re.GetReplacementString( "\2 == \1" );

Will give:

repl == "Kelly == wyrdrune.com!kelly";

As an implementation note: the CRegExp version of a similarly named function returned a newly allocated pointer array. Whilst this is efficient, it puts the onus upon the user of the class to delete it (correctly, with delete [] ) after it’s done with. Considering how the reference counting is implemented in the MFC CString class, passing CStrings around on the stack isn’t that expensive, the allocation only happens when the string data is initially allocated, with the ownership of the actual string data being handed from one CString instance to another as needed. Finally when the CString goes out of scope the data is deleted. This is efficient, and much more robust than having to keep track of which functions are allocators and which ones are not.

CString Regexp::GetErrorString() const;

Return a description of the most recent error caused on this Regexp. Errors include, but are not limited to, various forms of compilation errors, usually syntax errors, and calling Match when the Regexp hasn’t been initialized correctly (or at all). There are a fair number of these that should never occur if all of the Regexp use comes from your code, but where the user can type in regular expressions that you then have to compile, checking this can be very important.

bool Regexp::CompiledOK() const;

Regular Expression Syntax

A regular expression is zero or more branches, separated by '|'. It matches anything that matches one of the branches.

A branch is zero or more pieces, concatenated. It matches a match for the first, followed by a match for the second, etc.

A piece is an atom possibly followed by '*', '+', or '?'. An atom followed by '*' matches a sequence of 0 or more matches of the atom. An atom followed by '+' matches a sequence of 1 or more matches of the atom. An atom followed by '?' matches a match of the atom, or the null string.

An atom is a regular expression in parentheses (matching a match for the regular expression), a range (see below), '.' (matching any single character), '^' (matching the null string at the beginning of the input string), '$' (matching the null string at the end of the input string), a '\' followed by a single character (matching that character), or a single character with no other significance (matching that character).

A range is a sequence of characters enclosed in '[]'. It normally matches any single character from the sequence. If the sequence begins with '^', it matches any single character not from the rest of the sequence. If two characters in the sequence are separated by '-', this is shorthand for the full list of ASCII characters between them (e.g. '[0-9]' matches any decimal digit). To include a literal ']' in the sequence, make it the first character (following a possible '^'). To include a literal '-', make it the first or last character.

Ambiguity

If a regular expression could match two different parts of the input string, it will match the one which begins earliest. If both begin in the same place but match different lengths, or match the same length in different ways, life gets messier, as follows.

In general, the possibilities in a list of branches are considered in left-to-right order, the possibilities for '*', '+', and '?' are considered longest-first, nested constructs are considered from the outermost in, and concatenated constructs are considered leftmost-first. The match that will be chosen is the one that uses the earliest possibility in the first choice that has to be made. If there is more than one choice, the next will be made in the same manner (earliest possibility) subject to the decision on the first choice. And so forth.

For example, '(ab|a)b*c' could match 'abc' in one of two ways. The first choice is between 'ab' and 'a'; since 'ab' is earlier, and does lead to a successful overall match, it is chosen. Since the 'b' is already spoken for, the 'b*' must match its last possibility--the empty string--since it must respect the earlier choice.

In the particular case where the regular expression does not use `|' and does not apply `*', `+', or `?' to parenthesized subexpressions, the net effect is that the longest possible match will be chosen. So `ab*', presented with `xabbbby', will match `abbbb'. Note that if `ab*' is tried against `xabyabbbz', it will match `ab' just after `x', due to the begins-earliest rule. (In effect, the decision on where to start the match is the first choice to be made, hence subsequent choices must respect it even if this leads them to less-preferred alternatives.)

The Source

The accompanying archive contains the regexp library, as well as two separate test programs.

The first (originally enough called Test1) is a C++ port of the original test program that came with the C code. I’ve updated it to use the C++ constructs that the new library exposes. It acts as a useful sanity check and regression test when I’ve been modifying the source.

The second test is much simpler and uses the libraries substring extraction function to chop fields out of an email header, this is less of a test program and more of a simple sample.

Download Source.

A Note about Character Size

This code (and the samples) work and have been tested pretty thoroughly under Single Byte Character Sets (SBCS) and UNICODE. It will NOT work under Multi Byte Character Sets (MBCS), though it will compile which is very misleading. The problem (for anyone interested in fixing it) is that the internal representation of the ‘program’ requires a fixed size character, it manipulates this using memcpy() and memmove() without any knowledge of whether a particular element in it’s array is some internal code or a character. Making this use variable width characters would be a real pain since much more of the code would have to decode the program itself in order to determine whether a specific point in the program was looking at a operator or part of a character. Certainly this is doable, but it is more work than I want right now. The code works under UNICODE and that’s good enough for me. BTW even if the code is compiled with _MBCS it will only fail when it’s actually presented with multi-byte text, it’ll work just fine with 8-bit ASCII.