Skip to content

lit_deltalake.datasets.iterable_dataset

source module lit_deltalake.datasets.iterable_dataset

Classes

  • DeltaIterableDataset — Stream transformed rows from a Delta table through a reader implementation.

source class DeltaIterableDataset(table_uri: str, reader: DeltaReader, scan: DeltaScan | None = None, column_specs: tuple[ColumnSpec, ...] | None = None, row_transform: Transform | None = None, row_type: RowType | None = None, transform: Transform | None = None, shuffle_buffer_size: int = 0, shuffle_seed: int | None = None)

Bases : BaseDeltaDataset[SampleT], IterableDataset[SampleT]

Stream transformed rows from a Delta table through a reader implementation.

Create a streaming dataset for a Delta table.

Parameters

  • table_uri : str — URI of the Delta table.

  • reader : DeltaReader — Reader used to scan the table.

  • scan : DeltaScan | None — Optional configuration for the table snapshot.

  • column_specs : tuple[ColumnSpec, ...] | None — Optional mappings from source columns to sample fields.

  • row_transform : Transform | None — Optional transform applied to each selected row.

  • row_type : RowType | None — Optional dataclass type used to construct samples.

  • transform : Transform | None — Optional transform applied to each selected row.

  • shuffle_buffer_size : int — Maximum number of rows held for streaming shuffle.

  • shuffle_seed : int | None — Optional seed for streaming shuffle.

Raises

  • InvalidDatasetConfigurationError — If dataset configuration is invalid.

Methods

  • set_epoch — Set epoch used to derive the seeded streaming shuffle order.

  • transform_shard — Convert one scanned Arrow read batch into samples for this shard.

source method DeltaIterableDataset.set_epoch(epoch: int) → None

Set epoch used to derive the seeded streaming shuffle order.

Parameters

  • epoch : int — Non-negative epoch number.

Raises

  • ValueError — If epoch is negative.

source method DeltaIterableDataset.transform_shard(read_batch: pa.RecordBatch, shard_id: int, shard_count: int, row_index: int) → Iterator[SampleT]

Convert one scanned Arrow read batch into samples for this shard.

Readers with supports_source_sharding=True have already limited read_batch to this shard. Otherwise, this method applies row-level round-robin sharding using row_index. Override it to customize read-batch-level conversion while retaining the reader's sharding contract.