StreamTokenizer

Although

The

class is used to break any

into a sequence of

Consider

a program to count the occurrence of words in a text file:

//: SortedWordCount.java
// Counts words in a file, outputs
// results in sorted form.
import java.io.*;
import java.util.*;
import c08.*; // Contains StrSortVector
 
class Counter {
  private int i = 1;
  int read() { return i; }
  void increment() { i++; }
}
 
public class SortedWordCount {
  private FileInputStream file;
  private StreamTokenizer st;
  private Hashtable counts = new Hashtable();
  SortedWordCount(String filename)
    throws FileNotFoundException {
    try {
      file = new FileInputStream(filename);
      st = new StreamTokenizer(file);
      st.ordinaryChar('.');
      st.ordinaryChar('-');
    } catch(FileNotFoundException e) {
      System.out.println(
        "Could not open " + filename);
      throw e;
    }
  }
  void cleanup() {
    try {
      file.close();
    } catch(IOException e) {
      System.out.println(
        "file.close() unsuccessful");
    }
  }
  void countWords() {
    try {
      while(st.nextToken() !=
        StreamTokenizer.TT_EOF) {
        String s;
        switch(st.ttype) {
          case StreamTokenizer.TT_EOL:
            s = new String("EOL");
            break;
          case StreamTokenizer.TT_NUMBER:
            s = Double.toString(st.nval);
            break;
          case StreamTokenizer.TT_WORD:
            s = st.sval; // Already a String
            break;
          default: // single character in ttype
            s = String.valueOf((char)st.ttype);
        }
        if(counts.containsKey(s))
          ((Counter)counts.get(s)).increment();
        else
          counts.put(s, new Counter());
      }
    } catch(IOException e) {
      System.out.println(
        "st.nextToken() unsuccessful");
    }
  }
  Enumeration values() {
    return counts.elements();
  }
  Enumeration keys() { return counts.keys(); }
  Counter getCounter(String s) {
    return (Counter)counts.get(s);
  }
  Enumeration sortedKeys() {
    Enumeration e = counts.keys();
    StrSortVector sv = new StrSortVector();
    while(e.hasMoreElements())
      sv.addElement((String)e.nextElement());
    // This call forces a sort:
    return sv.elements();
  }
  public static void main(String[] args) {
    try {
      SortedWordCount wc =
        new SortedWordCount(args[0]);
      wc.countWords();
      Enumeration keys = wc.sortedKeys();
      while(keys.hasMoreElements()) {
        String key = (String)keys.nextElement();
        System.out.println(key + ": "
                 + wc.getCounter(key).read());
      }
      wc.cleanup();
    } catch(Exception e) {
      e.printStackTrace();
    }
  }
} ///:~ 

It

makes sense to present these in a sorted form, but since Java 1.0

To

open the file, a

is used, and to turn the file into words a

In

,

the tokens are pulled one at a time from the stream, and the

information is used to determine what to do with each token, since a token can

be an end-of-line, a number, a string, or a single character.

Once

a token is found, the

is not a type of

,

so it wasn’t inherited. It performs a specific type of functionality, so

even though the

and

methods must be re-exposed, that still doesn’t mean that

In

you can see the use of a

to open and count the words in a file – it just takes two lines of code.

Then an enumeration to a sorted list of keys (words) is extracted, and this is

used to pull out each key and associated

.

Note that the call to

is necessary to ensure that the file is closed.

A

second example using

can be found in Chapter 17.

StringTokenizer

Although

it isn’t part of the IO library, the

has sufficiently similar functionality to

that it will be described here.

The

You

ask a

object for the next token in the string using the

method, which either returns the token or an empty string to indicate that no

tokens remain.

As

an example, the following program performs a limited analysis of a sentence,

looking for key phrase sequences to indicate whether happiness or sadness is

implied.

//: AnalyzeSentence.java
// Look for particular sequences
// within sentences.
import java.util.*;
 
public class AnalyzeSentence {
  public static void main(String[] args) {
    analyze("I am happy about this");
    analyze("I am not happy about this");
    analyze("I am not! I am happy");
    analyze("I am sad about this");
    analyze("I am not sad about this");
    analyze("I am not! I am sad");
    analyze("Are you happy about this?");
    analyze("Are you sad about this?");
    analyze("It's you! I am happy");
    analyze("It's you! I am sad");
  }
  static StringTokenizer st;
  static void analyze(String s) {
    prt("nnew sentence >> " + s);
    boolean sad = false;
    st = new StringTokenizer(s);
    while (st.hasMoreTokens()) {
      String token = next();
      // Look until you find one of the
      // two starting tokens:
      if(!token.equals("I") &&
         !token.equals("Are"))
        continue; // Top of while loop
      if(token.equals("I")) {
        String tk2 = next();
        if(!tk2.equals("am")) // Must be after I
          break; // Out of while loop
        else {
          String tk3 = next();
          if(tk3.equals("sad")) {
            sad = true;
            break; // Out of while loop
          }
          if (tk3.equals("not")) {
            String tk4 = next();
            if(tk4.equals("sad"))
              break; // Leave sad false
            if(tk4.equals("happy")) {
              sad = true;
              break;
            }
          }
        }
      }
      if(token.equals("Are")) {
        String tk2 = next();
        if(!tk2.equals("you"))
          break; // Must be after Are
        String tk3 = next();
        if(tk3.equals("sad"))
          sad = true;
        break; // Out of while loop
      }
    }
    if(sad) prt("Sad detected");
  }
  static String next() {
    if(st.hasMoreTokens()) {
      String s = st.nextToken();
      prt(s);
      return s;
    }
    else
      return "";
  }
  static void prt(String s) {
    System.out.println(s);
  }
} ///:~ 

For

each string being analyzed, a

loop is entered and tokens are pulled off the string. Notice the first

statement, which says to

(go back to the beginning of the loop and start again) if the token is neither

an “I” nor an “Are.”

This

means that it will get tokens until an “I” or an “Are”

is found. You might think to use the

instead of the

The

logic of the rest of the

method is that the pattern that’s being searched for is “I am

sad,” “I am not happy,” or “Are you sad?” Without

the

statement, the code for this would be even messier than it is. You should be

aware that a typical parser (this is a primitive example of one) normally has a

table of these tokens and a piece of code that moves through the states in the

table as new tokens are read.

You

should think of the

only as shorthand for a simple and specific kind of

.

However, if you have a

that you want to tokenize and

is too limited, all you have to do is turn it into a stream with

and then use that to create a much more powerful

.

|

|