public static class StatefulTokenizer.Token extends Object
Constructor and Description |
---|
StatefulTokenizer.Token(int start,
int end,
Object type,
String content)
Initializes a new token with absolute start and end position, a type
and text content.
|
Modifier and Type | Method and Description |
---|---|
void |
append(StatefulTokenizer.Token t)
Joins two tokens by appending the contents of another token to this
token.
|
String |
getContent()
Returns the content of the token.
|
int |
getEnd()
Returns the absolute position where the token ends in the input
stream.
|
int |
getStart()
Returns the absolute position where the token starts in the input
stream.
|
Object |
getType()
Returns the type of the token.
|
String |
toString() |
public StatefulTokenizer.Token(int start, int end, Object type, String content)
start
- Absolute position where the token started in the input stream.end
- Absolute position where the token ended in the input stream.type
- Type of the token as defined by the corresponding rule.content
- The relevant text content from the input stream.public void append(StatefulTokenizer.Token t)
t
- Another token that should be appended to this tokenpublic int getStart()
public int getEnd()
public Object getType()
public String getContent()
Copyright © 2009-2013. All Rights Reserved.