Skip to content

lit_deltalake.readers.base

source module lit_deltalake.readers.base

Classes

source dataclass ReaderCapabilities(supports_worker_processes: bool = True, supports_keyed_lookup: bool = False, exact_length: bool = False, supports_source_sharding: bool = False)

Operations supported by a Delta reader implementation.

supports_source_sharding is false when scan_shard cannot restrict the source query to assigned data, as with readers that collect a full Spark or Sail query. In that case, the dataset applies row-level sharding after scanning each batch.

source class DeltaReader()

Bases : ABC

Read a Delta table scan as Arrow record batches.

Methods

  • scan — Yield Arrow record batches for a configured Delta table scan.

  • scan_shard — Yield record batches assigned to one distributed shard.

  • length — Return exact scan cardinality when it is cheap and reliable.

source method DeltaReader.scan(request: DeltaScan) → Iterator[pa.RecordBatch]

Yield Arrow record batches for a configured Delta table scan.

source method DeltaReader.scan_shard(request: DeltaScan, shard_id: int, shard_count: int) → Iterator[pa.RecordBatch]

Yield record batches assigned to one distributed shard.

Readers without source-level sharding fall back to a full scan; callers must apply row-level sharding to that result.

source method DeltaReader.length(request: DeltaScan) → int | None

Return exact scan cardinality when it is cheap and reliable.