Skip to content

lit_deltalake.readers

source package lit_deltalake.readers

Classes

  • DeltaReader — Read a Delta table scan as Arrow record batches.

  • DeltaRsReader — Read Delta table snapshots through delta-rs and PyArrow.

  • ReaderCapabilities — Operations supported by a Delta reader implementation.

  • SailDeltaReader — Read Delta tables through a caller-owned Sail Spark Connect session.

  • SparkDeltaReader — Read Delta tables from a caller-owned Spark or Spark Connect session.

Functions

source class DeltaReader()

Bases : ABC

Read a Delta table scan as Arrow record batches.

Methods

  • scan — Yield Arrow record batches for a configured Delta table scan.

  • scan_shard — Yield record batches assigned to one distributed shard.

  • length — Return exact scan cardinality when it is cheap and reliable.

source method DeltaReader.scan(request: DeltaScan) → Iterator[pa.RecordBatch]

Yield Arrow record batches for a configured Delta table scan.

source method DeltaReader.scan_shard(request: DeltaScan, shard_id: int, shard_count: int) → Iterator[pa.RecordBatch]

Yield record batches assigned to one distributed shard.

Readers without source-level sharding fall back to a full scan; callers must apply row-level sharding to that result.

source method DeltaReader.length(request: DeltaScan) → int | None

Return exact scan cardinality when it is cheap and reliable.

source class DeltaRsReader()

Bases : DeltaReader

Read Delta table snapshots through delta-rs and PyArrow.

Methods

  • scan — Yield Arrow record batches for a Delta-rs table scan.

  • scan_shard — Yield batches from fragments deterministically assigned to one shard.

  • length — Return the exact number of rows selected by a Delta-rs scan.

source method DeltaRsReader.scan(request: DeltaScan) → Iterator['pa.RecordBatch']

Yield Arrow record batches for a Delta-rs table scan.

Parameters

  • request : DeltaScan — Configuration for the table snapshot and selected data.

source method DeltaRsReader.scan_shard(request: DeltaScan, shard_id: int, shard_count: int) → Iterator[pa.RecordBatch]

Yield batches from fragments deterministically assigned to one shard.

Raises

  • ValueError

source method DeltaRsReader.length(request: DeltaScan) → int

Return the exact number of rows selected by a Delta-rs scan.

source dataclass ReaderCapabilities(supports_worker_processes: bool = True, supports_keyed_lookup: bool = False, exact_length: bool = False, supports_source_sharding: bool = False)

Operations supported by a Delta reader implementation.

supports_source_sharding is false when scan_shard cannot restrict the source query to assigned data, as with readers that collect a full Spark or Sail query. In that case, the dataset applies row-level sharding after scanning each batch.

source class SailDeltaReader(session: SparkSession | None = None)

Bases : SparkDeltaReader

Read Delta tables through a caller-owned Sail Spark Connect session.

Create a Sail-backed Delta reader.

Parameters

  • session : SparkSession | None — Optional caller-owned Spark or Spark Connect session.

source class SparkDeltaReader(session: SparkSession | None = None)

Bases : DeltaReader

Read Delta tables from a caller-owned Spark or Spark Connect session.

Create a Spark-backed Delta reader.

Parameters

  • session : SparkSession | None — Optional caller-owned Spark or Spark Connect session.

Methods

  • scan — Yield Arrow record batches for a Spark table scan.

source method SparkDeltaReader.scan(request: DeltaScan) → Iterator['pa.RecordBatch']

Yield Arrow record batches for a Spark table scan.

Parameters

  • request : DeltaScan — Configuration for the table snapshot and selected data.

Raises

  • InvalidDatasetConfigurationError

  • UnsupportedReaderOperationError

source start_sail_spark_connect_server(port: int = 50051) → SparkConnectServer

Start a PySail-backed Spark Connect server in the foreground.