Skip to content

lit_deltalake.dataloaders

source package lit_deltalake.dataloaders

Classes

Functions

source create_pytorch_dataloader(table_uri: str, columns: tuple[str, ...] | None = None, *, column_specs: tuple[ColumnSpec, ...] | None = None, filters: tuple[Filter, ...] = (), version: int | None = None, storage_options: StorageOptions | None = None, scan_batch_size: int = 65536, row_transform: Transform | None = None, row_type: RowType | None = None, transform: Transform | None = None, shuffle_buffer_size: int = 0, shuffle_seed: int | None = None, batch_size: int = 32, num_workers: int = 0, pin_memory: bool = False, persistent_workers: bool = False, prefetch_factor: int | None = None, multiprocessing_context: str | None = None, drop_last: bool = False, batch_format: Literal['default', 'tensordict'] = 'default', reader: DeltaReader | None = None, backend: Literal['delta-rs', 'spark', 'sail'] = 'delta-rs', spark_session: Any | None = None) → DeltaDataLoader[SampleT]

Create a PyTorch DataLoader backed by a Delta table scan.

Rows are sharded across initialized PyTorch distributed ranks and DataLoader workers by :class:DeltaIterableDataset. Use shuffle_buffer_size for bounded streaming shuffle; PyTorch sampler shuffle is unsupported for iterable datasets. Spark and Sail readers collect data to client memory; use the default Delta-rs backend for multi-rank DDP input.

Raises

  • InvalidDatasetConfigurationError

source class DeltaDataLoader(dataset: DeltaIterableDataset[SampleT], batch_size: int | None = 1, shuffle: bool | None = None, sampler: Any = None, batch_sampler: Any = None, num_workers: int = 0, collate_fn: Callable[[list[SampleT]], object] | None = None, pin_memory: bool = False, drop_last: bool = False, timeout: float = 0, worker_init_fn: Callable[[int], None] | None = None, multiprocessing_context: str | None = None, generator: object = None, *, prefetch_factor: int | None = None, persistent_workers: bool = False, pin_memory_device: str = '', in_order: bool = True, batch_format: Literal['default', 'tensordict'] = 'default')

Bases : DataLoader[SampleT]

Load batches from a Delta iterable dataset.

Create a DataLoader with Delta-specific batching support.

Parameters

  • dataset : DeltaIterableDataset[SampleT] — Iterable dataset that provides Delta table samples.

  • batch_size : int | None — Number of samples per batch.

  • shuffle : bool | None — Whether to shuffle samples using the PyTorch DataLoader API.

  • sampler : Any — Optional sampler for selecting samples.

  • batch_sampler : Any — Optional sampler that yields batches of indices.

  • num_workers : int — Number of worker processes used to load data.

  • collate_fn : Callable[[list[SampleT]], object] | None — Function used to combine samples into a batch.

  • pin_memory : bool — Whether to copy batches into pinned memory.

  • drop_last : bool — Whether to drop the final incomplete batch.

  • timeout : float — Maximum time in seconds to wait for a worker response.

  • worker_init_fn : Callable[[int], None] | None — Optional function called when each worker starts.

  • multiprocessing_context : str | None — Multiprocessing context used by workers.

  • generator : object — Optional random generator used by the loader.

  • prefetch_factor : int | None — Number of batches loaded in advance by each worker.

  • persistent_workers : bool — Whether to keep workers alive between iterations.

  • pin_memory_device : str — Device used for pinned memory.

  • in_order : bool — Whether to yield batches in worker order.

  • batch_format : Literal['default', 'tensordict'] — Batch collation format.