public abstract class CharTokenizer extends Tokenizer
Constructor and Description |
---|
CharTokenizer(java.io.Reader input) |
Modifier and Type | Method and Description |
---|---|
protected abstract boolean |
isTokenChar(char c)
Returns true iff a character should be included in a token.
|
Token |
next(Token reusableToken)
Returns the next token in the stream, or null at EOS.
|
protected char |
normalize(char c)
Called on each token character to normalize it before it is added to the
token.
|
void |
reset(java.io.Reader input)
Expert: Reset the tokenizer to a new reader.
|
next, reset
protected abstract boolean isTokenChar(char c)
protected char normalize(char c)
public final Token next(Token reusableToken) throws java.io.IOException
TokenStream
This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):
Token.clear()
before setting the fields in it & returning itTokenFilter
is considered a consumer.next
in class TokenStream
reusableToken
- a Token that may or may not be used to
return; this parameter should never be null (the callee
is not required to check for null before using it, but it is a
good idea to assert that it is not null.)java.io.IOException
public void reset(java.io.Reader input) throws java.io.IOException
Tokenizer
Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.