API Reference / preprocessing / text /

cleaners.py

Text cleaning and normalization utilities.

Provides preprocessing functions to clean and normalize text before tokenization and vectorization.

Classes

TextCleaner

class preprocessing.text.cleaners.TextCleaner(Transformer)

Comprehensive text cleaning transformer.

Applies multiple cleaning operations in sequence.
Constructor 
__init__(
    self,
    lowercase: bool = True,
    remove_punctuation: bool = False,
    remove_numbers: bool = False,
    remove_whitespace: bool = True,
    remove_html: bool = True,
    remove_urls: bool = True,
    remove_emails: bool = True,
    remove_special_chars: bool = False,
    strip_accents: bool = False,
    min_word_length: int = 1,
)

Parameters

lowercase
bool = True
Convert to lowercase.
remove_punctuation
bool = False
Remove all punctuation.
remove_numbers
bool = False
Remove all numbers.
remove_whitespace
bool = True
Normalize whitespace (multiple spaces to single).
remove_html
bool = True
Remove HTML tags.
remove_urls
bool = True
Remove URLs.
remove_emails
bool = True
Remove email addresses.
remove_special_chars
bool = False
Remove special characters.
strip_accents
bool = False
Remove accent marks from characters.
min_word_length
int = 1
Remove words shorter than this.
python 
>>> from tuiml.preprocessing.text import TextCleaner
>>>
>>> cleaner = TextCleaner(
...     lowercase=True,
...     remove_html=True,
...     remove_urls=True
... )
>>> clean_text = cleaner.transform(["<p>Visit https://example.com!</p>"])
>>> print(clean_text[0])  # 'visit'

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

fit (self, X, y=None) -> 'TextCleaner'

Fit (no-op for cleaner).

transform (self, texts: List[str]) -> List[str]

Clean a list of text documents.

Parameters
texts
list of str
Input texts.
Returns
cleaned
list of str
Cleaned texts.
__repr__ (self) -> str

StopWordRemover

class preprocessing.text.cleaners.StopWordRemover(Transformer)

Remove stop words from text.

Constructor 
__init__(
    self,
    stop_words: str | List[str] = 'english',
    case_sensitive: bool = False,
)

Parameters

stop_words
list of str or 'english' = 'english'
Stop words to remove.
case_sensitive
bool = False
Whether matching is case-sensitive.
python 
>>> from tuiml.preprocessing.text import StopWordRemover
>>> remover = StopWordRemover(stop_words='english')
>>> remover.transform(["the cat sat on the mat"])
['cat sat mat']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

fit (self, X, y=None) -> 'StopWordRemover'

Fit (no-op).

transform (self, texts: List[str]) -> List[str]

Remove stop words from texts.

__repr__ (self) -> str

Stemmer

class preprocessing.text.cleaners.Stemmer(Transformer)

Apply stemming to reduce words to their root form.

Implements Porter Stemmer algorithm.
Constructor 
__init__(
    self,
    lowercase: bool = True,
)

Parameters

lowercase
bool = True
Convert to lowercase before stemming.
python 
>>> from tuiml.preprocessing.text import Stemmer
>>> stemmer = Stemmer()
>>> stemmer.transform(["running cats are playing"])
['run cat are play']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

fit (self, X, y=None) -> 'Stemmer'

Fit (no-op).

transform (self, texts: List[str]) -> List[str]

Stem all words in texts.

__repr__ (self) -> str

TextNormalizer

class preprocessing.text.cleaners.TextNormalizer(Transformer)

Normalize text with multiple operations.

Constructor 
__init__(
    self,
    form: str = 'NFKC',
    lowercase: bool = True,
    strip: bool = True,
    collapse_whitespace: bool = True,
)

Parameters

form
str = 'NFKC'
Unicode normalization form ('NFC', 'NFD', 'NFKC', 'NFKD').
lowercase
bool = True
Convert to lowercase.
strip
bool = True
Strip leading/trailing whitespace.
collapse_whitespace
bool = True
Replace multiple whitespace with single space.
python 
>>> from tuiml.preprocessing.text import TextNormalizer
>>> normalizer = TextNormalizer()
>>> normalizer.transform(["  Hello   World  "])
['hello world']

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for parameters.

fit (self, X, y=None) -> 'TextNormalizer'

Fit (no-op).

transform (self, texts: List[str]) -> List[str]

Normalize texts.

__repr__ (self) -> str