lit_deltalake

source package lit_deltalake

Classes

• DeltaDataModule — Create stage-specific Delta datasets and their DataLoaders.

source class DeltaDataModule[SampleT](train_factory: DatasetFactory[SampleT] | None = None, validation_factory: DatasetFactory[SampleT] | None = None, test_factory: DatasetFactory[SampleT] | None = None, batch_size: int = 32, num_workers: int = 0, pin_memory: bool = False, persistent_workers: bool = False, prefetch_factor: int | None = None, multiprocessing_context: str | None = None, drop_last: bool = False, batch_format: Literal['default', 'tensordict'] = 'default', train_loader_config: DeltaDataLoaderConfig | None = None, validation_loader_config: DeltaDataLoaderConfig | None = None, test_loader_config: DeltaDataLoaderConfig | None = None)

Bases : lightning.LightningDataModule

Create stage-specific Delta datasets and their DataLoaders.

Create a Lightning data module for stage-specific Delta datasets.

Parameters

• train_factory : DatasetFactory[SampleT] | None — Factory that creates the training dataset.

• validation_factory : DatasetFactory[SampleT] | None — Factory that creates the validation dataset.

• test_factory : DatasetFactory[SampleT] | None — Factory that creates the test dataset.

• batch_size : int — Number of samples per batch.

• num_workers : int — Number of worker processes used to load data.

• pin_memory : bool — Whether to copy batches into pinned memory.

• persistent_workers : bool — Whether to keep workers alive between iterations.

• prefetch_factor : int | None — Number of batches loaded in advance by each worker.

• multiprocessing_context : str | None — Multiprocessing context used by workers.

• drop_last : bool — Whether to drop the final incomplete batch.

• batch_format : Literal['default', 'tensordict'] — Batch collation format.

• train_loader_config : DeltaDataLoaderConfig | None — Optional training DataLoader overrides.

• validation_loader_config : DeltaDataLoaderConfig | None — Optional validation DataLoader overrides.

• test_loader_config : DeltaDataLoaderConfig | None — Optional test DataLoader overrides.

Attributes

• hparams : Union[AttributeDict, MutableMapping] — meth:save_hyperparameters. It is mutable by the user. For the frozen set of initial hyperparameters, use :attr:hparams_initial.

• hparams_initial : AttributeDict — meth:save_hyperparameters. These contents are read-only. Manual updates to the saved hyperparameters can instead be performed through :attr:hparams.

Methods

• setup — Create datasets required for the requested training stage.

• train_dataloader — Return the training DataLoader.

• val_dataloader — Return the validation DataLoader.

• test_dataloader — Return the test DataLoader.

source method DeltaDataModule.setup(stage: str | None = None) → None

Create datasets required for the requested training stage.

Parameters

• stage : str | None — Stage to prepare, or None to prepare all stages.

source method DeltaDataModule.train_dataloader() → DataLoader[SampleT]

Return the training DataLoader.

source method DeltaDataModule.val_dataloader() → DataLoader[SampleT]

Return the validation DataLoader.

source method DeltaDataModule.test_dataloader() → DataLoader[SampleT]

Return the test DataLoader.