|
| 1 | +# Kratos Wheel Generation: Understanding PyProject files |
| 2 | + |
| 3 | +## TLDR |
| 4 | + |
| 5 | +Here is a quick summary of how to generate Kratos wheels depending on your deployment target: |
| 6 | + |
| 7 | +### 1. Generating a local release from the build dir |
| 8 | +After building Kratos with CMake, simply navigate to your newly generated release folder (which will contain the `libs` sub-folder with your `.so`/`.dll` files) and run the standard python build module from there: |
| 9 | +```bash |
| 10 | +cd bin/Release/ |
| 11 | +python3 -m build --wheel |
| 12 | +``` |
| 13 | + |
| 14 | +### 2. Generating a bundled release (Unified Wheel) |
| 15 | +To build a single "fat" wheel containing the Kratos Core alongside all compiled Applications, use the provided package orchestration script. Inside `scripts/wheels/build_wheel.py`, ensure the flag `CURRENT_CONFIG['UNIFIED_WHEEL'] = True`, then execute it from the project root: |
| 16 | +```bash |
| 17 | +python3 scripts/wheels/build_wheel.py |
| 18 | +``` |
| 19 | + |
| 20 | +### 3. Generating a regular release (Different wheels for Core and Apps) |
| 21 | +To build an ecosystem of separate wheels (one main wheel for the Core, and one standalone wheel per Application), ensure exactly the opposite: `CURRENT_CONFIG['UNIFIED_WHEEL'] = False` (which is typically the default). Run the orchestrator: |
| 22 | +```bash |
| 23 | +python3 scripts/wheels/build_wheel.py |
| 24 | +``` |
| 25 | + |
| 26 | +## Introduction |
| 27 | + |
| 28 | +In Kratos Multiphysics, wheel packaging is governed by several `pyproject.toml` files using standard PEP-517 tools with `hatchling` as the build system. This document outlines the purpose and critical features of the `pyproject.toml` files found in the core and scripts directories. |
| 29 | + |
| 30 | +## Context: Target Execution Path |
| 31 | + |
| 32 | +> [!WARNING] |
| 33 | +> While these files are physically authored at `kratos/pyproject.toml` and `scripts/wheels/pyproject.toml` (or similar), **they are not executed from these source locations**. |
| 34 | +
|
| 35 | +During the CMake configuration and build step, the required `pyproject.toml` files are orchestrated and copied to build-staging directories or final release binary directories. Wheels must be built from these target staging locations, as that is where all the compiled dynamic libraries (`.so` / `.dll`) and Python wrappers securely reside at build time. |
| 36 | + |
| 37 | +## 1. Core vs Unified Wheel Generators |
| 38 | + |
| 39 | +Kratos defines distinct strategies to package either the minimal core or a "fat" unified package containing all applications: |
| 40 | + |
| 41 | +### The Core `pyproject.toml` (`./kratos`) |
| 42 | +**Purpose:** This configuration generates a wheel corresponding exclusively to the Kratos kernel. It bundles only the core files, the MPI core wrappers, and fundamental system orchestrators. |
| 43 | +**Location in Source:** `kratos/pyproject.toml` |
| 44 | + |
| 45 | +### The Applications `pyproject.toml` (`./applications/[APP_NAME]/pyproject.toml`) |
| 46 | +**Purpose:** This configuration generates a wheel corresponding to the selected application. It is ment to be used along with the core pyproject.toml file to generate application wheels that depend on the core wheel. |
| 47 | +**Location in Source:** `kratos/pyproject.toml` |
| 48 | + |
| 49 | +### The Unified `pyproject.toml` (`./scripts`) |
| 50 | +**Purpose:** This project builds Kratos and **all compiled applications** from your local environment as a single bundled package. |
| 51 | +**Location in Source:** `scripts/wheels/pyproject.toml` |
| 52 | + |
| 53 | +## 2. `pyproject.toml` Structure |
| 54 | + |
| 55 | +Both configs share a common anatomy dictated by the `hatchling` backend, but handle certain specifics differently. |
| 56 | + |
| 57 | +### Standard Metadata (`[build-system]` and `[project]`) |
| 58 | +Defines Kratos Multiphysics as the package name, the backend used (`hatch.build`), target python versions, and dependencies: |
| 59 | +```toml |
| 60 | +[build-system] |
| 61 | +requires = ["hatchling"] |
| 62 | +build-backend = "hatchling.build" |
| 63 | + |
| 64 | +[project] |
| 65 | +name = "KratosMultiphysics" |
| 66 | +dependencies = ["numpy>=1.20"] |
| 67 | +dynamic = ["version"] |
| 68 | +``` |
| 69 | + |
| 70 | +### The Kratos Specific Section `[kratos]` (Core only) |
| 71 | +Because `kratos/pyproject.toml` handles the core natively alongside application builds, it contains a `[kratos]` table explicitly defining the C++ libraries needed for the kernel build. |
| 72 | + |
| 73 | +```toml |
| 74 | +[kratos] |
| 75 | +libs = [ |
| 76 | + "Kratos.*", |
| 77 | + "KratosMPI.*", |
| 78 | + "KratosCore.*", |
| 79 | + # ... |
| 80 | +] |
| 81 | +``` |
| 82 | +> [!NOTE] |
| 83 | +> This section is parsed by the Kratos wrapping orchestration script `build_wheel.py` to filter precisely which binaries belong to the core package versus applications and is not part of a standard pyproject.toml file. The unified pyproject omits this because it intentionally scoops up *everything* in the binary tree. |
| 84 | +
|
| 85 | +### Build Hooks `[tool.hatch.build.targets.wheel.hooks.custom]` |
| 86 | +Kratos wheels are strictly platform-specific and are not purely python code. These hooks execute a local `hatch_build.py` script which forces the final wheel to be recognized as system-specific. |
| 87 | +The hook script typically sets: |
| 88 | +- `build_data["infer_tag"] = True` |
| 89 | +- `build_data["pure_pyton"] = False` |
| 90 | + |
| 91 | +This ensures the wheel filenames correspond exactly to the current OS architecture (e.g., `linux_x86_64` or `win_amd64`) instead of `any`. It also injects the `KRATOS_VERSION` environment variable. |
| 92 | + |
| 93 | +## 3. `.py`, `.dll`, and `.so` Files |
| 94 | + |
| 95 | +### Python source and Hinting files inclusion |
| 96 | +Python files native to the framework are injected explicitly via standard `include` directives. |
| 97 | +```toml |
| 98 | +[tool.hatch.build.targets.wheel] |
| 99 | +include = [ |
| 100 | + "KratosMultiphysics/**/*.py", |
| 101 | + "KratosMultiphysics/**/*.pyi" |
| 102 | +] |
| 103 | +``` |
| 104 | + |
| 105 | +### Including Compiled C++ Artifacts (`.dll` / `.so`) |
| 106 | +Packaging `.dll` (Windows), `.so` (Linux) and `.dylib` (Mac) files side-by-side with Python code is structurally the most critical operation of Kratos wheels. Kratos relies on `hatchling`'s `force-include` mapping technique mapping: |
| 107 | + |
| 108 | +```toml |
| 109 | +[tool.hatch.build.targets.wheel.force-include] |
| 110 | +libs = "KratosMultiphysics/.libs" |
| 111 | +``` |
| 112 | + |
| 113 | +**How it works:** |
| 114 | +1. A staging folder named `libs` is set up containing all required `.so` or `.dll` binaries. |
| 115 | +2. The key string (`libs`) in the TOML corresponds to this physical layout. |
| 116 | +3. The assigned value (`"KratosMultiphysics/.libs"`) forces `hatchling` to place the files into that destination directory structure inside the final `.whl` archive. |
| 117 | +4. Hence, all collected compiled `.so`/`.dll`/`.dylib` packages are pushed into the hidden `.libs` directory, correctly satisfying dynamic links at runtime. |
| 118 | + |
| 119 | +--- |
| 120 | + |
| 121 | +## 4. The `build_wheel.py` Script |
| 122 | + |
| 123 | +While it is possible to build wheels manually by fulfilling the `force-include` prerequisites, Kratos provides a robust automation script to handle this: `scripts/wheels/build_wheel.py`. |
| 124 | + |
| 125 | +### Purpose |
| 126 | +The `build_wheel.py` script is the main driver for generating Kratos Python Wheels. It orchestrates the CMake build of Kratos Python interfaces and coordinates the PEP-517 wheel packaging across multiple operating systems and Python versions. |
| 127 | + |
| 128 | +### Execution Flow & Usage |
| 129 | + |
| 130 | +**1. Managing the Build Target (Unified vs Separate Packages)** |
| 131 | +The script contains a configuration rule `CURRENT_CONFIG['UNIFIED_WHEEL']`: |
| 132 | +- **If `False` (Packages Target):** The script sets up the build for the "Core" using `kratos/pyproject.toml`. It then dynamically detects all compiled directories inside `applications/*Application*` and generates a standalone `.whl` extension package for *each application*. |
| 133 | +- **If `True` (Unified Target):** The script executes a single package generation utilizing `scripts/wheels/pyproject.toml`. This creates one massive package containing both the core and all application libraries. |
| 134 | + |
| 135 | +**2. Staging Environment (`setupWheelDir`)** |
| 136 | +It dynamically generates an isolated `WHEEL_ROOT` staging directory. For each iteration, the script copies: |
| 137 | +- The compiled Python wrappers (`.py` files). |
| 138 | +- The raw binary libraries (`.dll` / `.so` mapped to a folder called `libs`). |
| 139 | +- The relevant `pyproject.toml`. |
| 140 | +- The `README.md` and `hatch_build.py` hook. |
| 141 | + |
| 142 | +**3. Binary Filtering Using `[kratos]` Config** |
| 143 | +Because applications and core share a master binary release folder (`bin/Release/libs`), the script performs intelligent filtering. |
| 144 | +It uses Python's `toml.load` to read the target `pyproject.toml`. If the file contains a `[kratos] libs = [...]` list (like `kratos/pyproject.toml`), the orchestration script aggressively iterates over the physics binaries within the staging `libs` directory and **deletes any binaries that do not match the declared patterns**. |
| 145 | +This strictly ensures that the Core wheel gets only Core binaries, and Application wheels get only their respective subsets. |
| 146 | + |
| 147 | +**4. Executing Hatchling & Wheel Output** |
| 148 | +After staging and filtering, the orchestrator triggers the Python build module natively inside the temporary root: |
| 149 | +```python |
| 150 | +subprocess.run([python_interpreter, "-m", "build", "--wheel", ...]) |
| 151 | +``` |
| 152 | +Resulting `.whl` files are exported to the target `--outdir` (often `WHEEL_OUT`), and the staging configuration gets safely destroyed via `cleanupWheelDir`. |
0 commit comments