dbx_container.engine

source module dbx_container.engine

Classes

• RuntimeContainerEngine — Engine for building container variations across all Databricks runtimes.

source class RuntimeContainerEngine(data_dir: Path | str = Path('./data'), max_workers: int = 5, verify_ssl: bool = False, latest_lts_count: int | None = 2, force_ubuntu_version: str | None = None, skip_ml_variants: bool = True)

Engine for building container variations across all Databricks runtimes.

Initialize the ContainerEngine.

Parameters

• data_dir : Path | str — Directory where to save generated files

• max_workers : int — Maximum number of worker threads for scraping

• verify_ssl : bool — Whether to verify SSL certificates when making HTTP requests

• latest_lts_count : int | None — Number of latest LTS versions to build (None for all LTS versions, default: 2)

• force_ubuntu_version : str | None — Force a specific Ubuntu version for all base images (e.g., "22.04"). If None, defaults to "24.04" unless matching runtime's OS version.

• skip_ml_variants : bool — Skip ML runtime variants during generation (default: True)

Methods

• get_dependency_image_reference — Get the image reference for the dependency of a given image type.

• should_upgrade_os_version — Check if OS version should be upgraded to default (24.04).

• get_python_versions_from_runtime — Extract Python version information from runtime to configure containers.

• sanitize_runtime_version — Sanitize runtime version by replacing whitespace with dashes.

• extract_os_version_from_runtime — Extract OS version information from runtime.

• get_runtime_variations — Get all OS and Python version variations for a runtime.

• generate_dockerfile_for_image_type — Generate a Dockerfile for a specific image type and runtime.

• save_dockerfile — Save a generated Dockerfile to the appropriate location.

• generate_requirements_txt — Generate requirements.txt from runtime's included libraries.

• save_runtime_metadata — Save runtime metadata as JSON for reference.

• build_all_images_for_runtime — Build all image variations for a single runtime.

• build_non_runtime_specific_images — Build image types that don't need runtime variations.

• save_runtime_metadata_generic — Save generic runtime metadata for non-runtime-specific images.

• build_all_images_for_all_runtimes — Build all image variations for all available runtimes.

• save_build_summary — Save a summary of all generated files.

• generate_build_matrix — Generate a GitHub Actions build matrix from the build summary.

• run — Main entry point to run the complete engine process.

source method RuntimeContainerEngine.get_dependency_image_reference(image_type: str, runtime: Runtime | None = None, variation: dict[str, str] | None = None, registry: str | None = None, use_gpu_base: bool = False) → str

Get the image reference for the dependency of a given image type.

Parameters

• image_type : str — The image type to get the dependency for

• runtime : Runtime | None — The runtime (if runtime-specific)

• variation : dict[str, str] | None — The variation config (if applicable)

• registry : str | None — Registry prefix (e.g., 'ghcr.io/owner/dbx-runtime'). If None, uses local tag.

• use_gpu_base : bool — Whether to use GPU variant base images

Returns

• str — Full image reference to use as FROM base

source method RuntimeContainerEngine.should_upgrade_os_version(variation: dict[str, str] | None) → tuple[bool, str]

Check if OS version should be upgraded to default (24.04).

Parameters

• variation : dict[str, str] | None — The variation config with os_version

Returns

• tuple[bool, str] — Tuple of (should_upgrade, os_version_to_use)

source method RuntimeContainerEngine.get_python_versions_from_runtime(runtime: Runtime) → PythonDockerfileVersions

Extract Python version information from runtime to configure containers.

Parameters

• runtime : Runtime — The runtime object containing environment information

Returns

• PythonDockerfileVersions — PythonDockerfileVersions object with extracted version info

source staticmethod RuntimeContainerEngine.sanitize_runtime_version(version: str) → str

Sanitize runtime version by replacing whitespace with dashes.

Parameters

• version : str — The runtime version string (e.g., "15.4 LTS")

Returns

• str — Sanitized version string (e.g., "15.4-LTS")

source method RuntimeContainerEngine.extract_os_version_from_runtime(runtime: Runtime) → str

Extract OS version information from runtime.

Parameters

• runtime : Runtime — The runtime object containing environment information

Returns

• str — Ubuntu version string (e.g., "22.04", "24.04")

source method RuntimeContainerEngine.get_runtime_variations(runtime: Runtime) → list[dict[str, str]]

Get all OS and Python version variations for a runtime.

Parameters

• runtime : Runtime — The runtime object

Returns

• list[dict[str, str]] — List of variation configurations with os_version and python_version

source method RuntimeContainerEngine.generate_dockerfile_for_image_type(runtime: Runtime, image_type: str, config: dict[str, Any], variation: dict[str, str] | None = None, registry: str | None = None) → str

Generate a Dockerfile for a specific image type and runtime.

Parameters

• runtime : Runtime — The runtime to build the container for

• image_type : str — The type of image to build

• config : dict[str, Any] — Configuration for the image type

• variation : dict[str, str] | None — Optional variation config with os_version and python_version

• registry : str | None — Optional registry prefix for image naming

Returns

• str — The generated Dockerfile content as a string

source method RuntimeContainerEngine.save_dockerfile(dockerfile_content: str, runtime: Runtime, image_type: str, variation: dict[str, str] | None = None) → Path

Save a generated Dockerfile to the appropriate location.

Parameters

• dockerfile_content : str — The Dockerfile content to save

• runtime : Runtime — The runtime this Dockerfile is for

• image_type : str — The type of image

• variation : dict[str, str] | None — Optional variation config for naming

Returns

• Path — Path to the saved file

source method RuntimeContainerEngine.generate_requirements_txt(runtime: Runtime, image_type: str, variation: dict[str, str] | None = None) → Path

Generate requirements.txt from runtime's included libraries.

Parameters

• runtime : Runtime — The runtime with included_libraries data

• image_type : str — The type of image (should be python or python-gpu)

• variation : dict[str, str] | None — Optional variation config for naming

Returns

• Path — Path to the generated requirements.txt file

source method RuntimeContainerEngine.save_runtime_metadata(runtime: Runtime, image_type: str, variation: dict[str, str] | None = None) → Path

Save runtime metadata as JSON for reference.

Parameters

• runtime : Runtime — The runtime to save metadata for

• image_type : str — The type of image this metadata corresponds to

• variation : dict[str, str] | None — Optional variation config for naming

Returns

• Path — Path to the saved metadata file

source method RuntimeContainerEngine.build_all_images_for_runtime(runtime: Runtime, registry: str | None = None) → dict[str, list[Path]]

Build all image variations for a single runtime.

Parameters

• runtime : Runtime — The runtime to build images for

• registry : str | None — Optional registry prefix for image naming

Returns

• dict[str, list[Path]] — Dictionary mapping image types to lists of generated file paths

source method RuntimeContainerEngine.build_non_runtime_specific_images(registry: str | None = None) → dict[str, list[Path]]

Build image types that don't need runtime variations.

Parameters

• registry : str | None — Optional registry prefix for image naming

Returns

• dict[str, list[Path]] — Dictionary mapping image types to lists of generated file paths

source method RuntimeContainerEngine.save_runtime_metadata_generic(runtime: Runtime, image_type: str) → Path

Save generic runtime metadata for non-runtime-specific images.

Parameters

• runtime : Runtime — The reference runtime used for metadata

• image_type : str — The type of image this metadata corresponds to

Returns

• Path — Path to the saved metadata file

source method RuntimeContainerEngine.build_all_images_for_all_runtimes(registry: str | None = None) → dict[str, dict[str, list[Path]]]

Build all image variations for all available runtimes.

Parameters

• registry : str | None — Optional registry prefix for image naming

Returns

• Nested dictionary — {runtime_version: {image_type: [file_paths]}}

source method RuntimeContainerEngine.save_build_summary(all_generated_files: dict[str, dict[str, list[Path]]]) → Path

Save a summary of all generated files.

Parameters

• all_generated_files : dict[str, dict[str, list[Path]]] — The complete mapping of generated files

Returns

• Path — Path to the saved summary file

source method RuntimeContainerEngine.generate_build_matrix(only_lts: bool = False, image_type: str | None = None, latest_lts_count: int | None = None) → dict

Generate a GitHub Actions build matrix from the build summary.

Parameters

• only_lts : bool — If True, only include LTS runtimes

• image_type : str | None — If specified, only include this image type

• latest_lts_count : int | None — If specified, only include the N latest LTS versions

Returns

• dict — Dictionary with matrix configuration for GitHub Actions

source method RuntimeContainerEngine.run(registry: str | None = None) → dict[str, dict[str, list[Path]]]

Main entry point to run the complete engine process.

Parameters

• registry : str | None — Optional registry prefix for image naming

Returns

• dict[str, dict[str, list[Path]]] — Dictionary mapping runtime versions to generated files