Skip to content

lit_deltalake.dataloaders.dataloader

source module lit_deltalake.dataloaders.dataloader

Classes

source class DeltaDataLoader(dataset: DeltaIterableDataset[SampleT], batch_size: int | None = 1, shuffle: bool | None = None, sampler: Any = None, batch_sampler: Any = None, num_workers: int = 0, collate_fn: Callable[[list[SampleT]], object] | None = None, pin_memory: bool = False, drop_last: bool = False, timeout: float = 0, worker_init_fn: Callable[[int], None] | None = None, multiprocessing_context: str | None = None, generator: object = None, *, prefetch_factor: int | None = None, persistent_workers: bool = False, pin_memory_device: str = '', in_order: bool = True, batch_format: Literal['default', 'tensordict'] = 'default')

Bases : DataLoader[SampleT]

Load batches from a Delta iterable dataset.

Create a DataLoader with Delta-specific batching support.

Parameters

  • dataset : DeltaIterableDataset[SampleT] — Iterable dataset that provides Delta table samples.

  • batch_size : int | None — Number of samples per batch.

  • shuffle : bool | None — Whether to shuffle samples using the PyTorch DataLoader API.

  • sampler : Any — Optional sampler for selecting samples.

  • batch_sampler : Any — Optional sampler that yields batches of indices.

  • num_workers : int — Number of worker processes used to load data.

  • collate_fn : Callable[[list[SampleT]], object] | None — Function used to combine samples into a batch.

  • pin_memory : bool — Whether to copy batches into pinned memory.

  • drop_last : bool — Whether to drop the final incomplete batch.

  • timeout : float — Maximum time in seconds to wait for a worker response.

  • worker_init_fn : Callable[[int], None] | None — Optional function called when each worker starts.

  • multiprocessing_context : str | None — Multiprocessing context used by workers.

  • generator : object — Optional random generator used by the loader.

  • prefetch_factor : int | None — Number of batches loaded in advance by each worker.

  • persistent_workers : bool — Whether to keep workers alive between iterations.

  • pin_memory_device : str — Device used for pinned memory.

  • in_order : bool — Whether to yield batches in worker order.

  • batch_format : Literal['default', 'tensordict'] — Batch collation format.