lit_deltalake.datamodule

Classes

DeltaDataLoaderConfig — DataLoader settings for a data module stage.
DeltaDataModule — Create stage-specific Delta datasets and their DataLoaders.

source dataclass DeltaDataLoaderConfig(batch_size: int = 1, num_workers: int = 0, pin_memory: bool = False, persistent_workers: bool = False, prefetch_factor: int | None = None, multiprocessing_context: str | None = None, drop_last: bool = False, batch_format: Literal['default', 'tensordict'] = 'default')

DataLoader settings for a data module stage.

source class DeltaDataModule[SampleT](train_factory: DatasetFactory[SampleT] | None = None, validation_factory: DatasetFactory[SampleT] | None = None, test_factory: DatasetFactory[SampleT] | None = None, batch_size: int = 32, num_workers: int = 0, pin_memory: bool = False, persistent_workers: bool = False, prefetch_factor: int | None = None, multiprocessing_context: str | None = None, drop_last: bool = False, batch_format: Literal['default', 'tensordict'] = 'default', train_loader_config: DeltaDataLoaderConfig | None = None, validation_loader_config: DeltaDataLoaderConfig | None = None, test_loader_config: DeltaDataLoaderConfig | None = None)

Bases : lightning.LightningDataModule

Create stage-specific Delta datasets and their DataLoaders.

Create a Lightning data module for stage-specific Delta datasets.

Parameters

train_factory : DatasetFactory[SampleT] | None — Factory that creates the training dataset.
validation_factory : DatasetFactory[SampleT] | None — Factory that creates the validation dataset.
test_factory : DatasetFactory[SampleT] | None — Factory that creates the test dataset.
batch_size : int — Number of samples per batch.
num_workers : int — Number of worker processes used to load data.
pin_memory : bool — Whether to copy batches into pinned memory.
persistent_workers : bool — Whether to keep workers alive between iterations.
prefetch_factor : int | None — Number of batches loaded in advance by each worker.
multiprocessing_context : str | None — Multiprocessing context used by workers.
drop_last : bool — Whether to drop the final incomplete batch.
batch_format : Literal['default', 'tensordict'] — Batch collation format.
train_loader_config : DeltaDataLoaderConfig | None — Optional training DataLoader overrides.
validation_loader_config : DeltaDataLoaderConfig | None — Optional validation DataLoader overrides.
test_loader_config : DeltaDataLoaderConfig | None — Optional test DataLoader overrides.

Attributes

hparams : Union[AttributeDict, MutableMapping] — meth:save_hyperparameters. It is mutable by the user. For the frozen set of initial hyperparameters, use :attr:hparams_initial.
hparams_initial : AttributeDict — meth:save_hyperparameters. These contents are read-only. Manual updates to the saved hyperparameters can instead be performed through :attr:hparams.

Methods

setup — Create datasets required for the requested training stage.
train_dataloader — Return the training DataLoader.
val_dataloader — Return the validation DataLoader.
test_dataloader — Return the test DataLoader.

source method DeltaDataModule.setup(stage: str | None = None) → None

Create datasets required for the requested training stage.

Parameters

stage : str | None — Stage to prepare, or None to prepare all stages.

source method DeltaDataModule.train_dataloader() → DataLoader[SampleT]

Return the training DataLoader.

source method DeltaDataModule.val_dataloader() → DataLoader[SampleT]

Return the validation DataLoader.

source method DeltaDataModule.test_dataloader() → DataLoader[SampleT]

Return the test DataLoader.