public abstract class Tokenizer extends TokenStream
This is an abstract class.
NOTE: subclasses must override TokenStream.next(Token)
. It's
also OK to instead override TokenStream.next()
but that
method is now deprecated in favor of TokenStream.next(Token)
.
NOTE: subclasses overriding TokenStream.next(Token)
must
call Token.clear()
.
Modifier and Type | Field and Description |
---|---|
protected java.io.Reader |
input
The text source for this Tokenizer.
|
Modifier | Constructor and Description |
---|---|
protected |
Tokenizer()
Construct a tokenizer with null input.
|
protected |
Tokenizer(java.io.Reader input)
Construct a token stream processing the given input.
|
protected Tokenizer()
protected Tokenizer(java.io.Reader input)
public void close() throws java.io.IOException
close
in class TokenStream
java.io.IOException
public void reset(java.io.Reader input) throws java.io.IOException
java.io.IOException
Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.