public class NGramTokenizer extends Tokenizer
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_NGRAM_SIZE |
static int |
DEFAULT_MIN_NGRAM_SIZE |
Constructor and Description |
---|
NGramTokenizer(Reader input)
Creates NGramTokenizer with default min and max n-grams.
|
NGramTokenizer(Reader input,
int minGram,
int maxGram)
Creates NGramTokenizer with given min and max n-grams.
|
public static final int DEFAULT_MIN_NGRAM_SIZE
public static final int DEFAULT_MAX_NGRAM_SIZE
public NGramTokenizer(Reader input, int minGram, int maxGram)
input
- Reader holding the input to be tokenizedminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic NGramTokenizer(Reader input)
input
- Reader holding the input to be tokenizedpublic final Token next(Token reusableToken) throws IOException
next
in class TokenStream
reusableToken
- a Token that may or may not be used to
return; this parameter should never be null (the callee
is not required to check for null before using it, but it is a
good idea to assert that it is not null.)IOException
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.