Skip to content

lit_deltalake.datasets.base

source module lit_deltalake.datasets.base

Classes

  • BaseDeltaDataset — Shared configuration and record conversion for Delta datasets.

source class BaseDeltaDataset[SampleT](reader: DeltaReader, scan: DeltaScan, column_specs: tuple[ColumnSpec, ...] | None, row_transform: Transform | None, row_type: RowType | None, transform: Transform | None)

Bases : ABC

Shared configuration and record conversion for Delta datasets.

Initialize shared Delta dataset configuration.

Parameters

  • reader : DeltaReader — Reader used to scan the table.

  • scan : DeltaScan — Configuration for the table snapshot.

  • column_specs : tuple[ColumnSpec, ...] | None — Optional mappings from source columns to sample fields.

  • row_transform : Transform | None — Optional transform applied to each selected row.

  • row_type : RowType | None — Optional dataclass type used to construct samples.

  • transform : Transform | None — Optional transform applied to each selected row.

Raises

  • InvalidDatasetConfigurationError — If shared configuration is invalid.

Methods

  • transform_record — Convert one Arrow-derived Python record into the configured sample.

source method BaseDeltaDataset.transform_record(record: Record) → SampleT

Convert one Arrow-derived Python record into the configured sample.

RecordBatch.to_pylist() has already converted the Arrow row into a Python mapping when this hook runs. Column decoders, column transforms, dataclass construction, and the whole-sample transform are applied here. Subclasses can override this method to customize per-record conversion.