API Reference / preprocessing / text /

vectorizers.py

Text vectorization utilities - Convert text to numeric feature vectors.

Classes

CountVectorizer

class preprocessing.text.vectorizers.CountVectorizer(Transformer)

Convert a collection of text documents to a matrix of token counts.

Implements a "bag-of-words" representation where each document is represented by the frequency of terms in the vocabulary.
Constructor 
__init__(
    self,
    tokenizer: BaseTokenizer = None,
    max_features: int = None,
    min_df: Union[int, float] = 1,
    max_df: float = 1.0,
    binary: bool = False,
    lowercase: bool = True,
    stop_words: Union[List[str], str] = None,
    ngram_range: tuple = (),
    vocabulary: Dict[str, int] = None,
)

Parameters

tokenizer
BaseTokenizer
The strategy for splitting text into tokens. If None, a WordTokenizer is used.
max_features
int
The maximum number of terms to include in the vocabulary. If provided, it keeps only the top max_features ordered by term frequency across the corpus.
min_df
int or float = 1

Minimum document frequency to include a term.

  • •Int: Minimum absolute count.
  • •Float: Minimum proportion of documents.
max_df
float = 1.0
Maximum proportion of documents a term can appear in to be included. Useful for filtering out corpus-specific stop words.
binary
bool = False
If True, all non-zero counts are set to 1. This is useful for discrete probabilistic models that only care about presence/absence.
lowercase
bool = True
If True, converts all text to lowercase before tokenization.
stop_words
list of str or 'english'
A list of words that will be filtered out. If "english", a built-in list is used.
ngram_range
tuple (min_n, max_n), 1) = (1
The lower and upper boundary of the range of n-values for different n-grams to be extracted.

Attributes

vocabulary_
dict
Mapping of terms to feature indices.
feature_names_
list of str
Ordered list of terms corresponding to the column indices.

See Also

TfidfVectorizer

Counts weighted by importance.
HashingVectorizer

Memory-efficient counting.

Vectorize a small corpus:

python 
>>> from tuiml.preprocessing.text import CountVectorizer
>>> docs = ["cat in the hat", "hat on the mat"]
>>> vectorizer = CountVectorizer(stop_words='english')
>>> X = vectorizer.fit_transform(docs)
>>> print(vectorizer.get_feature_names_out())
['cat', 'hat', 'mat']

Methods

get_parameter_schema (cls) -> Dict[str, Dict]

Return JSON Schema for parameters.

fit (self, documents: List[str], y=None) -> 'CountVectorizer'

Learn vocabulary from documents.

Parameters
documents
list of str
List of text documents.
y
ignored
Returns
self
transform (self, documents: List[str]) -> np.ndarray

Transform documents to count matrix.

Parameters
documents
list of str
Documents to transform.
Returns
X
ndarray of shape (n_documents, n_features)
Document-term matrix.
fit_transform (self, documents: List[str], y=None) -> np.ndarray

Fit and transform documents.

get_feature_names_out (self) -> List[str]

Get feature names.

__repr__ (self) -> str

TfidfTransformer

class preprocessing.text.vectorizers.TfidfTransformer(Transformer)

Transform a count matrix to a TF-IDF representation.

Constructor 
__init__(
    self,
    norm: str = 'l2',
    use_idf: bool = True,
    smooth_idf: bool = True,
    sublinear_tf: bool = False,
)

Overview

TF-IDF (Term Frequency-Inverse Document Frequency) balances the local importance of a term (TF) with its global rarity (IDF).

Theory

The IDF weight for a term t is calculated as:

\text{idf}(t) = \log \frac{n}{df(t)} + 1

where n is the total number of documents and df(t) is the number of documents containing term t.

Parameters

norm
{'l1', 'l2', None} = 'l2'

Normalization strategy for each row:

  • •"l2": Sum of squares is 1 (Euclidean norm).
  • •"l1": Sum of absolute values is 1.
use_idf
bool = True
If True, enables inverse document frequency reweighting.
smooth_idf
bool = True
If True, adds 1 to document frequencies to prevent zero division: :math:`\log \frac{n+1}{df+1} + 1`.
sublinear_tf
bool = False
If True, applies sublinear scaling to term frequency: :math:`1 + \log(\text{tf})`.

Attributes

idf_
ndarray
The learned inverse document frequency vector.

See Also

TfidfVectorizer

Combined counting and weighting.

Weight a count matrix:

python 
>>> from tuiml.preprocessing.text import TfidfTransformer
>>> import numpy as np
>>> counts = np.array([[3, 0, 1], [2, 1, 0]])
>>> transformer = TfidfTransformer()
>>> X_tfidf = transformer.fit_transform(counts)

Methods

get_parameter_schema (cls) -> Dict[str, Dict]

Return JSON Schema for parameters.

fit (self, X: np.ndarray, y=None) -> 'TfidfTransformer'

Learn IDF weights from count matrix.

Parameters
X
ndarray of shape (n_documents, n_features)
Document-term count matrix.
Returns
self
transform (self, X: np.ndarray) -> np.ndarray

Transform count matrix to TF-IDF.

Parameters
X
ndarray of shape (n_documents, n_features)
Document-term count matrix.
Returns
X_tfidf
ndarray
TF-IDF weighted matrix.
fit_transform (self, X: np.ndarray, y=None) -> np.ndarray

Fit and transform.

__repr__ (self) -> str

TfidfVectorizer

class preprocessing.text.vectorizers.TfidfVectorizer(Transformer)

Convert a collection of raw documents to a matrix of TF-IDF features.

Equivalent to CountVectorizer followed by TfidfTransformer.
Constructor 
__init__(
    self,
    tokenizer: BaseTokenizer = None,
    max_features: int = None,
    min_df: Union[int, float] = 1,
    max_df: float = 1.0,
    lowercase: bool = True,
    stop_words: Union[List[str], str] = None,
    ngram_range: tuple = (),
    norm: str = 'l2',
    use_idf: bool = True,
    smooth_idf: bool = True,
    sublinear_tf: bool = False,
)

Parameters

tokenizer
BaseTokenizer
The strategy for splitting text into tokens.
max_features
int
The maximum number of terms to include in the vocabulary.
min_df
int or float = 1
Minimum document frequency (see CountVectorizer).
max_df
float = 1.0
Maximum document frequency (see CountVectorizer).
lowercase
bool = True
If True, converts text to lowercase.
ngram_range
tuple, 1) = (1
The range of n-grams to extract.
norm
{'l1', 'l2', None} = 'l2'
Normalization strategy (see TfidfTransformer).
use_idf
bool = True
Enable IDF weighting.
smooth_idf
bool = True
Smooth IDF weights.
sublinear_tf
bool = False
Use sublinear TF scaling.

Attributes

vocabulary_
dict
Mapping of terms to feature indices.
idf_
ndarray
The learned inverse document frequency vector.

Directly compute TF-IDF features:

python 
>>> from tuiml.preprocessing.text import TfidfVectorizer
>>> docs = ["the cat", "the dog"]
>>> vectorizer = TfidfVectorizer()
>>> X = vectorizer.fit_transform(docs)
>>> print(X.shape)
(2, 3)

Methods

get_parameter_schema (cls) -> Dict[str, Dict]

Return JSON Schema for parameters.

fit (self, documents: List[str], y=None) -> 'TfidfVectorizer'

Learn vocabulary and IDF weights.

Parameters
documents
list of str
Training documents.
Returns
self
transform (self, documents: List[str]) -> np.ndarray

Transform documents to TF-IDF matrix.

Parameters
documents
list of str
Documents to transform.
Returns
X
ndarray of shape (n_documents, n_features)
TF-IDF matrix.
fit_transform (self, documents: List[str], y=None) -> np.ndarray

Fit and transform documents.

vocabulary_ (self) -> Dict[str, int]

Get vocabulary mapping.

idf_ (self) -> np.ndarray

Get IDF weights.

get_feature_names_out (self) -> List[str]

Get feature names.

__repr__ (self) -> str

HashingVectorizer

class preprocessing.text.vectorizers.HashingVectorizer(Transformer)

Convert text to a fixed-size feature matrix using the hashing trick.

Constructor 
__init__(
    self,
    n_features: int = ...,
    tokenizer: BaseTokenizer = None,
    lowercase: bool = True,
    stop_words: Union[List[str], str] = None,
    ngram_range: tuple = (),
    binary: bool = False,
    norm: str = 'l2',
)

Overview

HashingVectorizer is a memory-efficient alternative to CountVectorizer. It doesn't store a vocabulary in memory, but instead hashes terms directly to a fixed number of buckets.

Notes

Pros:
  • Very low memory footprint (does not store vocabulary).
  • Can handle large, out-of-core datasets.
Cons:
  • No inverse mapping (cannot retrieve the original words from indices).
  • Potential for hash collisions.

Parameters

n_features
int = 2**20
The number of bins in the hash table. Larger values reduce collisions but increase memory usage of output matrices.
tokenizer
BaseTokenizer
The strategy for splitting text into tokens.
lowercase
bool = True
If True, converts text to lowercase.
ngram_range
tuple, 1) = (1
The range of n-grams to extract.
binary
bool = False
If True, all non-zero counts are set to 1.
norm
{'l1', 'l2', None} = 'l2'
Normalization strategy for output vectors.

Memory-efficient vectorization:

python 
>>> from tuiml.preprocessing.text import HashingVectorizer
>>> docs = ["cat in the hat"]
>>> vectorizer = HashingVectorizer(n_features=128)
>>> X = vectorizer.fit_transform(docs)
>>> print(X.shape)
(1, 128)

Methods

get_parameter_schema (cls) -> Dict[str, Dict]

Return JSON Schema for parameters.

fit (self, documents: List[str], y=None) -> 'HashingVectorizer'

Fit (no-op for hashing vectorizer).

transform (self, documents: List[str]) -> np.ndarray

Transform documents to hashed feature matrix.

Parameters
documents
list of str
Documents to transform.
Returns
X
ndarray of shape (n_documents, n_features)
Hashed feature matrix.
fit_transform (self, documents: List[str], y=None) -> np.ndarray

Fit and transform.

__repr__ (self) -> str