API Reference / preprocessing / text /

tokenizers.py

Text tokenization utilities.

Classes

BaseTokenizer

class preprocessing.text.tokenizers.BaseTokenizer(ABC)

Base class for text tokenizers.

Methods

tokenize (self, text: str) -> List[str]

Tokenize a text string.

Parameters
text
str
Input text to tokenize.
Returns
tokens
list of str
List of tokens.
__call__ (self, text: str) -> List[str]

Allow tokenizer to be called directly.

WordTokenizer

class preprocessing.text.tokenizers.WordTokenizer(BaseTokenizer)

Simple word tokenizer that splits on whitespace and punctuation.

Constructor 
__init__(
    self,
    lowercase: bool = True,
    remove_punctuation: bool = True,
    min_length: int = 1,
)

Parameters

lowercase
bool = True
Convert tokens to lowercase.
remove_punctuation
bool = True
Remove punctuation from tokens.
min_length
int = 1
Minimum token length to keep.
python 
>>> from tuiml.preprocessing.text import WordTokenizer
>>> tokenizer = WordTokenizer()
>>> tokenizer.tokenize("Hello, World! This is a test.")
['hello', 'world', 'this', 'is', 'a', 'test']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

tokenize (self, text: str) -> List[str]

Tokenize text into words.

__repr__ (self) -> str

NGramTokenizer

class preprocessing.text.tokenizers.NGramTokenizer(BaseTokenizer)

N-gram tokenizer that generates character or word n-grams.

Constructor 
__init__(
    self,
    n: int = 2,
    max_n: int = None,
    level: str = 'word',
    lowercase: bool = True,
)

Parameters

n
int = 2
N-gram size.
max_n
int
Maximum n-gram size (for generating range of n-grams).
level
str = 'word'
'word' for word n-grams, 'char' for character n-grams.
lowercase
bool = True
Convert to lowercase first.
python 
>>> from tuiml.preprocessing.text import NGramTokenizer
>>>
>>> # Word bigrams
>>> tokenizer = NGramTokenizer(n=2, level='word')
>>> tokenizer.tokenize("the quick brown fox")
['the quick', 'quick brown', 'brown fox']
>>>
>>> # Character trigrams
>>> tokenizer = NGramTokenizer(n=3, level='char')
>>> tokenizer.tokenize("hello")
['hel', 'ell', 'llo']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

tokenize (self, text: str) -> List[str]

Generate n-grams from text.

__repr__ (self) -> str

RegexTokenizer

class preprocessing.text.tokenizers.RegexTokenizer(BaseTokenizer)

Tokenizer using regular expression patterns.

Constructor 
__init__(
    self,
    pattern: str = '\\w+',
    gaps: bool = False,
    lowercase: bool = True,
)

Parameters

pattern
str = r'
Regex pattern for matching tokens.
gaps
bool = False
If True, pattern matches gaps between tokens. If False, pattern matches tokens themselves.
lowercase
bool = True
Convert tokens to lowercase.
python 
>>> from tuiml.preprocessing.text import RegexTokenizer
>>>
>>> # Match words
>>> tokenizer = RegexTokenizer(pattern=r'\w+')
>>> tokenizer.tokenize("Hello, World!")
['hello', 'world']
>>>
>>> # Split on whitespace (gaps=True)
>>> tokenizer = RegexTokenizer(pattern=r'\s+', gaps=True)
>>> tokenizer.tokenize("Hello World")
['Hello', 'World']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

tokenize (self, text: str) -> List[str]

Tokenize using regex pattern.

__repr__ (self) -> str

SentenceTokenizer

class preprocessing.text.tokenizers.SentenceTokenizer(BaseTokenizer)

Tokenizer that splits text into sentences.

Constructor 
__init__(
    self,
    abbreviations: List[str] = None,
)

Parameters

abbreviations
list of str
Custom abbreviations to not split on (e.g., ['mr.', 'dr.']).
python 
>>> from tuiml.preprocessing.text import SentenceTokenizer
>>> tokenizer = SentenceTokenizer()
>>> tokenizer.tokenize("Hello world. How are you? I'm fine!")
['Hello world.', 'How are you?', "I'm fine!"]

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

tokenize (self, text: str) -> List[str]

Split text into sentences.

__repr__ (self) -> str

WhitespaceTokenizer

class preprocessing.text.tokenizers.WhitespaceTokenizer(BaseTokenizer)

Simple tokenizer that splits on whitespace only.

Constructor 
__init__(
    self,
    lowercase: bool = False,
)

Parameters

lowercase
bool = False
Convert tokens to lowercase.
python 
>>> from tuiml.preprocessing.text import WhitespaceTokenizer
>>> tokenizer = WhitespaceTokenizer()
>>> tokenizer.tokenize("Hello, World!")
['Hello,', 'World!']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

tokenize (self, text: str) -> List[str]

Split on whitespace.

__repr__ (self) -> str

TreebankTokenizer

class preprocessing.text.tokenizers.TreebankTokenizer(BaseTokenizer)

Penn Treebank style tokenizer.

Handles contractions, punctuation, and special cases following Penn Treebank conventions.
Constructor 
__init__(
    self,
)
python 
>>> from tuiml.preprocessing.text import TreebankTokenizer
>>> tokenizer = TreebankTokenizer()
>>> tokenizer.tokenize("They'll save and invest more.")
['They', "'ll", 'save', 'and', 'invest', 'more', '.']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

__init__ (self)
tokenize (self, text: str) -> List[str]

Tokenize using Treebank conventions.

__repr__ (self) -> str