protected static class StatefulTokenizer.Rule
extends java.lang.Object
getToken(String,int)
can be used to process input
data: If the rule matches a token is returned, otherwise null
will be returned.Constructor and Description |
---|
StatefulTokenizer.Rule(java.lang.String pattern,
java.lang.Object tokenType)
Initializes a new instance with the specified regular expression
pattern, a type for generated tokens.
|
StatefulTokenizer.Rule(java.lang.String pattern,
java.lang.Object tokenType,
java.lang.String nextState)
Initializes a new instance with the specified regular expression
pattern, a type for generated tokens, and a name of a grammar state
that should be triggered.
|
Modifier and Type | Method and Description |
---|---|
StatefulTokenizer.Token |
getToken(java.lang.String data,
int pos)
Analyzes the specified input data starting at the given position
and returns a token with the defined type, the content matched by
the regular expression if the rule matches.
|
public StatefulTokenizer.Rule(java.lang.String pattern, java.lang.Object tokenType, java.lang.String nextState)
pattern
- A regular expression pattern string.tokenType
- The type for the tokens generated by this rule.nextState
- The grammar state that should be used next to
analyze the input data if the rule matched.public StatefulTokenizer.Rule(java.lang.String pattern, java.lang.Object tokenType)
pattern
- A regular expression pattern string.tokenType
- The type for the tokens generated by this rule.public StatefulTokenizer.Token getToken(java.lang.String data, int pos)
null
will be returned.data
- Input data.pos
- Position to start looking for a match.null
if the rule didn't match.