Skip to content

lit_deltalake.dataloaders.pytorch

source module lit_deltalake.dataloaders.pytorch

Functions

source create_pytorch_dataloader(table_uri: str, columns: tuple[str, ...] | None = None, *, column_specs: tuple[ColumnSpec, ...] | None = None, filters: tuple[Filter, ...] = (), version: int | None = None, storage_options: StorageOptions | None = None, scan_batch_size: int = 65536, row_transform: Transform | None = None, row_type: RowType | None = None, transform: Transform | None = None, shuffle_buffer_size: int = 0, shuffle_seed: int | None = None, batch_size: int = 32, num_workers: int = 0, pin_memory: bool = False, persistent_workers: bool = False, prefetch_factor: int | None = None, multiprocessing_context: str | None = None, drop_last: bool = False, batch_format: Literal['default', 'tensordict'] = 'default', reader: DeltaReader | None = None, backend: Literal['delta-rs', 'spark', 'sail'] = 'delta-rs', spark_session: Any | None = None) → DeltaDataLoader[SampleT]

Create a PyTorch DataLoader backed by a Delta table scan.

Rows are sharded across initialized PyTorch distributed ranks and DataLoader workers by :class:DeltaIterableDataset. Use shuffle_buffer_size for bounded streaming shuffle; PyTorch sampler shuffle is unsupported for iterable datasets. Spark and Sail readers collect data to client memory; use the default Delta-rs backend for multi-rank DDP input.

Raises

  • InvalidDatasetConfigurationError