A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers. More...
#include <zorba/tokenizer.h>
Public Types | |
typedef Tokenizer::size_type | value_type |
Public Member Functions | |
State () | |
Default constructor. More... | |
Public Attributes | |
value_type | para |
Paragraph number. More... | |
value_type | sent |
Sentence number. More... | |
value_type | token |
Token number. More... | |
A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers.
Definition at line 54 of file tokenizer.h.
Definition at line 55 of file tokenizer.h.
zorba::Tokenizer::State::State | ( | ) |
Default constructor.
value_type zorba::Tokenizer::State::para |
Paragraph number.
Definition at line 59 of file tokenizer.h.
value_type zorba::Tokenizer::State::sent |
Sentence number.
Definition at line 58 of file tokenizer.h.
value_type zorba::Tokenizer::State::token |
Token number.
Definition at line 57 of file tokenizer.h.