Skip to content

lit_deltalake.datasets

source package lit_deltalake.datasets

Classes

  • BaseDeltaDataset — Shared configuration and record conversion for Delta datasets.

  • DeltaIterableDataset — Stream transformed rows from a Delta table through a reader implementation.

  • DeltaKeyDataset — Map explicit stable keys to individual Delta table rows.

source class BaseDeltaDataset[SampleT](reader: DeltaReader, scan: DeltaScan, column_specs: tuple[ColumnSpec, ...] | None, row_transform: Transform | None, row_type: RowType | None, transform: Transform | None)

Bases : ABC

Shared configuration and record conversion for Delta datasets.

Initialize shared Delta dataset configuration.

Parameters

  • reader : DeltaReader — Reader used to scan the table.

  • scan : DeltaScan — Configuration for the table snapshot.

  • column_specs : tuple[ColumnSpec, ...] | None — Optional mappings from source columns to sample fields.

  • row_transform : Transform | None — Optional transform applied to each selected row.

  • row_type : RowType | None — Optional dataclass type used to construct samples.

  • transform : Transform | None — Optional transform applied to each selected row.

Raises

  • InvalidDatasetConfigurationError — If shared configuration is invalid.

Methods

  • transform_record — Convert one Arrow-derived Python record into the configured sample.

source method BaseDeltaDataset.transform_record(record: Record) → SampleT

Convert one Arrow-derived Python record into the configured sample.

RecordBatch.to_pylist() has already converted the Arrow row into a Python mapping when this hook runs. Column decoders, column transforms, dataclass construction, and the whole-sample transform are applied here. Subclasses can override this method to customize per-record conversion.

source class DeltaIterableDataset(table_uri: str, reader: DeltaReader, scan: DeltaScan | None = None, column_specs: tuple[ColumnSpec, ...] | None = None, row_transform: Transform | None = None, row_type: RowType | None = None, transform: Transform | None = None, shuffle_buffer_size: int = 0, shuffle_seed: int | None = None)

Bases : BaseDeltaDataset[SampleT], IterableDataset[SampleT]

Stream transformed rows from a Delta table through a reader implementation.

Create a streaming dataset for a Delta table.

Parameters

  • table_uri : str — URI of the Delta table.

  • reader : DeltaReader — Reader used to scan the table.

  • scan : DeltaScan | None — Optional configuration for the table snapshot.

  • column_specs : tuple[ColumnSpec, ...] | None — Optional mappings from source columns to sample fields.

  • row_transform : Transform | None — Optional transform applied to each selected row.

  • row_type : RowType | None — Optional dataclass type used to construct samples.

  • transform : Transform | None — Optional transform applied to each selected row.

  • shuffle_buffer_size : int — Maximum number of rows held for streaming shuffle.

  • shuffle_seed : int | None — Optional seed for streaming shuffle.

Raises

  • InvalidDatasetConfigurationError — If dataset configuration is invalid.

Methods

  • set_epoch — Set epoch used to derive the seeded streaming shuffle order.

  • transform_shard — Convert one scanned Arrow read batch into samples for this shard.

source method DeltaIterableDataset.set_epoch(epoch: int) → None

Set epoch used to derive the seeded streaming shuffle order.

Parameters

  • epoch : int — Non-negative epoch number.

Raises

  • ValueError — If epoch is negative.

source method DeltaIterableDataset.transform_shard(read_batch: pa.RecordBatch, shard_id: int, shard_count: int, row_index: int) → Iterator[SampleT]

Convert one scanned Arrow read batch into samples for this shard.

Readers with supports_source_sharding=True have already limited read_batch to this shard. Otherwise, this method applies row-level round-robin sharding using row_index. Override it to customize read-batch-level conversion while retaining the reader's sharding contract.

source class DeltaKeyDataset(reader: DeltaReader, scan: DeltaScan, id_column: str, keys: Sequence[str | int], column_specs: tuple[ColumnSpec, ...] | None = None, row_transform: Transform | None = None, row_type: RowType | None = None, transform: Transform | None = None, cache_size: int = 1024)

Bases : BaseDeltaDataset[SampleT], Dataset[SampleT]

Map explicit stable keys to individual Delta table rows.

Create a map-style dataset for stable keyed Delta table rows.

Parameters

  • reader : DeltaReader — Reader used to scan the table.

  • scan : DeltaScan — Read configuration for the versioned table snapshot.

  • id_column : str — Source column containing each row's stable key.

  • keys : Sequence[str | int] — Stable keys identifying rows to retrieve.

  • column_specs : tuple[ColumnSpec, ...] | None — Optional mappings from source columns to sample fields.

  • row_transform : Transform | None — Optional transform applied to each selected row.

  • row_type : RowType | None — Optional dataclass type used to construct samples.

  • transform : Transform | None — Optional transform applied to each selected row.

  • cache_size : int — Maximum number of version-pinned records retained for repeated lookups.

Raises

  • InvalidDatasetConfigurationError — If dataset configuration is invalid.

  • UnsupportedReaderOperationError — If the reader does not support keyed lookup.