Apache Parquet reader and writer.
Parquet stores columns rather than rows, which keeps large datasets compact and makes reading a subset of columns cheap. Also handles partitioned directories, where the data is split across files by the value of one or more columns.
Requires pyarrow or fastparquet; both are imported lazily.
Functions
load_parquet(filepath: Union[str, Path], target_column: Optional[Union[str, int]]=..., columns: Optional[List[str]]=None, handle_categorical: str='encode', engine: str='auto') -> Dataset
Load data from Parquet file.
Parameters
filepath
str or Path
Path to Parquet file
target_column
str, int, or None
Column name or index for target variable (None for unsupervised, -1 for last column)
columns
list of str or None
List of columns to read (None for all)
handle_categorical
str
How to handle categorical columns:
- •'encode': Label encode to integers
- •'drop': Drop categorical columns
- •'error': Raise error if categorical found
engine
str
Parquet engine ('auto', 'pyarrow', 'fastparquet')
Returns
result
Dataset
Dataset object with X, y, feature_names
python
>>> from tuiml.datasets.loaders import load_parquet
>>> data = load_parquet('data.parquet', target_column='class')
>>> data.X.shape, data.y.shape
python
>>> # Read specific columns
>>> data = load_parquet('data.parquet', columns=['age', 'income', 'target'])save_parquet(filepath: Union[str, Path], data: np.ndarray, feature_names: Optional[List[str]]=None, target: Optional[np.ndarray]=None, target_names: Optional[List[str]]=None, target_column_name: str='target', compression: str='snappy', engine: str='auto')
Save data to Parquet format.
Parameters
filepath
str or Path
Output file path
data
numpy.ndarray
Feature data (n_samples, n_features)
feature_names
list of str or None
List of feature names
target
numpy.ndarray or None
Target values (optional)
target_names
list of str or None
Names of target classes (for classification)
target_column_name
str
Name for target column
compression
str or None
Compression codec ('snappy', 'gzip', 'brotli', None)
engine
str
Parquet engine ('auto', 'pyarrow', 'fastparquet')
python
>>> from tuiml.datasets.loaders import save_parquet
>>> save_parquet('output.parquet', X, feature_names=['a', 'b'], target=y)load_parquet_partitioned(directory: Union[str, Path], target_column: Optional[Union[str, int]]=..., filters: Optional[List]=None, handle_categorical: str='encode', engine: str='auto') -> Dataset
Load data from partitioned Parquet dataset.
Parameters
directory
str or Path
Path to partitioned parquet directory
target_column
str, int, or None
Column name or index for target variable
filters
list or None
Row group filters (e.g., [('col', '>', 5)])
handle_categorical
str
How to handle categorical columns
engine
str
Parquet engine
Returns
result
Dataset
Dataset object
python
>>> from tuiml.datasets.loaders import load_parquet_partitioned
>>> # Load partitioned dataset with filtering
>>> data = load_parquet_partitioned(
... 'data_partitioned/',
... filters=[('year', '>=', 2020)]
... )