From a18b18cc2de6ab732bd3e22290564e4926f8a864 Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Mon, 5 Oct 2026 17:51:18 -0700 Subject: [PATCH 1/2] Document Dragon and MPI execution architecture Explain worker placement, data movement and completion protocols. Separate the architecture overview from Dragon transport measurements. Signed-off-by: Trent Nelson --- docs/README.md | 11 +- docs/architecture.md | 2 + docs/components/xpois.md | 2 +- docs/distributed-execution.md | 23 +- docs/distributed.md | 172 +++++++++++ docs/dragon-performance.md | 212 +++++++++++++ docs/dragon.md | 552 +++++++++++++++++++--------------- docs/mpi.md | 127 ++++++++ 8 files changed, 840 insertions(+), 261 deletions(-) create mode 100644 docs/distributed.md create mode 100644 docs/dragon-performance.md create mode 100644 docs/mpi.md diff --git a/docs/README.md b/docs/README.md index efe2f3ca..fa7b8545 100644 --- a/docs/README.md +++ b/docs/README.md @@ -44,10 +44,15 @@ requirements. ## Adapting and validating - [Architecture](architecture.md): package boundaries and execution model. -- [Distributed execution](distributed-execution.md): prepare a pipeline +- [Distributed architecture](distributed.md): work assignment, GPU ownership, + data movement and completion. +- [Dragon architecture](dragon.md): process placement, GPU sharing with MPS, + queues and worker lifecycle. +- [MPI architecture](mpi.md): rank binding, collective coordination and file mode. +- [Distributed launch guide](distributed-execution.md): prepare a pipeline manifest and launch MPI or Dragon workers using Slurm or SSH host files. -- [Dragon transport and coordination](dragon.md): execution models, - consumer-local queues, transport selection, and timing boundaries. +- [Dragon transport and performance](dragon-performance.md): transport selection, + queue placement experiments and timing boundaries. - [Pipeline stage benchmark](components/pipeline-stage-benchmark.md): compare a persistent GPU pipeline with separate stages and inspect transfer costs. - [Adapting the workflows](adapting-workflows.md): a practical extension diff --git a/docs/architecture.md b/docs/architecture.md index 3720c5de..f43adca0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -112,6 +112,8 @@ Core's public CLI facade. Core also supplies FITS metadata inspection and reader selection, plus the shared Dragon/MPI execution machinery. Components retain their scientific input validation and choose how to divide work among those executors. +See the [distributed architecture](distributed.md) for process ownership, +data movement and the Dragon/MPI worker lifecycles. xDataReader, xFit, xPois, and xRep expose a curated Python surface for embedding numerical operations. xScan and xRay are primarily diff --git a/docs/components/xpois.md b/docs/components/xpois.md index b94c1a36..bab3674e 100644 --- a/docs/components/xpois.md +++ b/docs/components/xpois.md @@ -748,7 +748,7 @@ selection remains a launcher option; `--executor dragon` cannot change the transport of an already running Dragon session. The worker wall-time limit covers the entire invocation, including warmups and all measured rounds. -See [Dragon transport and coordination](../dragon.md) for native HSTA TCP +See [Dragon transport and performance](../dragon-performance.md) for native HSTA TCP launch options, consumer-local queues in the persistent executors, and measurement boundaries. diff --git a/docs/distributed-execution.md b/docs/distributed-execution.md index 4fb34158..521d9083 100644 --- a/docs/distributed-execution.md +++ b/docs/distributed-execution.md @@ -3,7 +3,9 @@ The xPois, xFit and xScan pipeline assigns complete image pairs to GPU workers. Each worker loads its model once and processes its assigned pairs through subtraction, dipole fitting and classification. MPI and Dragon -provide alternative launchers for the same workload. +provide alternative launchers for the same workload. The +[distributed architecture](distributed.md) explains work assignment, process +ownership and data movement. Three inputs determine a run: @@ -16,8 +18,8 @@ Three inputs determine a run: `mpiexec -n 8` requests eight processes. The host file or scheduler allocation determines where they run. With Dragon, `--nodes 2` selects two nodes and cuPhoton's `--max-workers 8` caps the total GPU workers. The examples below -use two nodes with four GPUs each, one worker per GPU, and at least eight -image-pair items. +use two nodes with four GPUs each and at least eight image-pair items. +They use the default placement of one worker per GPU. ## Prepare the environment and workload @@ -145,10 +147,16 @@ allocated nodes, and cuPhoton discovers their GPUs and places its workers: --max-workers 8 --warmup-rounds 1 --measure-rounds 3 ``` -`--max-workers` is a cap: Dragon also limits workers by available GPUs and -item count. This example selects TCP for both Dragon transports. Use the +`--max-workers` caps the total process count. With the default +`--workers-per-gpu 1`, Dragon also limits workers by available GPUs and item +count. This example selects TCP for both Dragon transports. Use the site's routable interface selection when needed; see -[Dragon transport and coordination](dragon.md). +[Dragon transport and performance](dragon-performance.md). + +For multiple workers per GPU, see [Dragon GPU sharing and MPS](dragon.md#share-a-gpu-between-workers) +and the [GPU sharing commands](components/xscan.md#share-a-gpu-between-image-pairs). +An explicit MPS connection requires an existing service on each worker host. +The launcher or administrator manages that service separately. ## Launch on named hosts through SSH @@ -227,7 +235,8 @@ Each command creates `runs//summary.json` and retains warmup and measured outputs under `rounds/warmup-*` and `rounds/measure-*`. Inspect the summary and worker records for actual host/GPU placement, successful item completion and cleanup. MPI rejects more ranks than image-pair items; -duplicate physical GPU assignments also fail validation. +unexpected sharing or a physical GPU identity that disagrees with the +requested placement also fails validation. Choose a new `--name` for another attempt. Readiness, batch execution, validation and cleanup have separate timing fields. Batch timing includes diff --git a/docs/distributed.md b/docs/distributed.md new file mode 100644 index 00000000..d6c92640 --- /dev/null +++ b/docs/distributed.md @@ -0,0 +1,172 @@ +# Distributed execution architecture + +cuPhoton uses Dragon or MPI to run independent work on several GPUs. Each +GPU worker is a Python process bound to one GPU. The worker loads its inputs, +runs the numerical code and writes its results. A coordinator assigns work +and checks completion. Dragon defaults to one worker per GPU. Its shared +executor can place several independent workers on the same GPU. + +The choice of Dragon or MPI controls process placement and coordination. The component +and numerical backend control the calculation. For example, the combined +imaging pipeline uses CuPy and PyTorch inside each worker with either runtime. + +Read [How cuPhoton uses Dragon](dragon.md) or [How cuPhoton uses MPI](mpi.md) +for the process and message details. The [distributed launch guide](distributed-execution.md) +contains environment setup, manifests and Slurm/SSH commands. + +## Units of work + +The optical workflows divide a batch into independent items: + +| Workflow | One item | +| --- | --- | +| xPois batch fitting | One aligned reference/target image pair | +| xFit batch fitting | A range of candidate stamps | +| xScan inference | A range of samples that preserves the planned minibatches | +| Combined xPois, xFit and xScan pipeline | One image pair with its candidate coordinates | + +An imaging worker completes all stages for an assigned pair on its assigned GPU. +The executors do not split that pair across GPUs or exchange intermediate +images between workers. The distributed xRay detector workflow has a separate +[tile and artifact contract](xray/DISTRIBUTED-DETECTOR-ARTIFACTS.md). + +A `WorkItem` contains an ID, a JSON description and a weight for assignment. +The description refers to files, array properties or sample ranges. It does +not contain a live GPU array. + +cuPhoton sorts items by descending weight and assigns each to the worker +with the smallest total weight. Each worker receives a fixed list, called +a *shard*. Weights usually estimate input bytes. xScan uses sample counts. +They do not predict execution time exactly. Workers process their shards +sequentially, with no work stealing. A round waits for its slowest worker. + +## Where the data goes + +This diagram shows the combined imaging pipeline inside one worker. The +other workers run the same stages on their own assigned pairs. + +```mermaid +flowchart TB + F["Shared files: images and model checkpoint"] + C["Coordinator"] + subgraph W["One Python worker process, one assigned GPU"] + R["Read and validate one image pair"] + P["xPois: match images and subtract"] + X["xFit: fit candidate stamps"] + S["xScan: classify candidates"] + H["Copy compact results to host memory"] + R --> P --> X --> S --> H + end + O["Shared files: results and completion records"] + F --> R + C -. "Launch reference and round control" .-> W + H --> O + W -. "Status, identity and timing" .-> C + O -->|"Read and validate"| C +``` + +Dragon passes a reference to a shared launch file. MPI ranks plan items +locally or receive them in a broadcast. + +Workers read inputs directly from the shared filesystem. NPY inputs pass +through host memory before a GPU upload. FITS inputs use the configured +reader, which can return host or device arrays. See the +[FITS reader guide](components/xdr.md#read-fits-images-in-a-workflow) for that choice. + +The combined pipeline keeps intermediate image and stamp arrays on the GPU. +CuPy and PyTorch share compatible device buffers through DLPack, an array +exchange interface. cuPhoton retains the buffer owners until the GPU work +finishes. These transfers occur within one process. Dragon and MPI carry +descriptions and completion metadata. They do not transport these GPU buffers. + +Workers write scientific outputs and completion records to shared storage. +The coordinator reads those records and checks the expected item IDs, +input identities and output files. The filesystem therefore carries both +bulk input/output and part of the completion protocol. + +## Process ownership + +| Responsibility | Dragon | MPI | +| --- | --- | --- | +| Start the application | Dragon launcher starts a coordinator within its runtime | MPI launcher starts all ranks | +| Start GPU workers | Coordinator creates a native Dragon `ProcessGroup` | Each rank is a GPU worker | +| Coordinate the run | Separate coordinator process | Rank 0, which also computes its shard | +| Select each GPU | Dragon host and GPU placement policy | Launcher binding and per-node rank mapping | +| Exchange status | Dragon native queues | MPI collectives; the xPois file mode uses shared files | + +Each worker must see one assigned GPU before a numerical library initializes +CUDA. Inside that process, the assigned GPU is device 0. cuPhoton records +the host, process ID and physical GPU identity. A CUDA ordinal alone does +not identify a physical GPU. + +MPI and the default Dragon configuration require distinct GPUs. With Dragon +sharing enabled, workers assigned to the same GPU must report the same +physical identity. Workers assigned to different GPUs must report different +identities. See [Dragon GPU sharing and MPS](dragon.md#share-a-gpu-between-workers) +for placement, connection checks and service ownership. + +Every node needs compatible software and access to the input, model and +output paths at the same absolute locations. The launch guide describes +these requirements and how to preserve scheduler GPU restrictions. + +## What workers retain + +In repeated runs, each GPU worker stays alive through all requested warmup +and measured rounds. A round repeats the same shards. Warmup rounds run real work and retain their outputs. + +| Worker | State retained between items or rounds | +| --- | --- | +| xPois repeated batch | Process and CUDA context; inputs are read again each round | +| xFit | Validated input, solver settings, model and CUDA context | +| xScan | Dataset, metadata rows, model and data loader | +| Combined pipeline | Model, CUDA streams and device context; each pair is read again | + +Each additional Dragon worker retains its own component state, including its +model where applicable. More workers can increase host and device memory use. +The pipeline also offers local process and thread executors; see the +[GPU sharing guide](components/xscan.md#share-a-gpu-between-image-pairs). + +Keeping a process alive does not keep every input image on the GPU. Output +writes, input checks and any required transfers still occur. Ordinary +single-pass xPois uses its own executor path. The Dragon and MPI pages +describe it. + +## Completion and failure + +The shared executors collect a completion record from each worker after a +round. They then validate the item records and run any component-specific +merge step. A record of success must agree with the files on disk. + +For a successful run, workers close their component resources before the +coordinator publishes the final `summary.json`. Failed work, missing records +or GPU identities that disagree with the requested placement make the run +fail. Failed shutdown also makes the run fail. A successful round alone does not +establish a successful complete run. + +These executors do not reschedule a failed item onto another worker. +Completed files remain available for diagnosis. Validation confirms one terminal record per expected item. It does not +retry or recover work. + +## Timing boundaries + +The reported `batch_wall_sec` covers round dispatch through worker completion +collection. Worker completion includes its item processing and record writes. +Coordinator validation and component merging follow this interval. The shared +executor reports component merging as `finalization_sec`. + +Measure launcher-to-exit wall time separately to include runtime startup and +process teardown. Keep warmup, readiness and shutdown costs visible when +comparing Dragon and MPI. Subtracting the longest worker time from batch time +does not isolate network latency. See +[transport measurements and timing](dragon-performance.md#evidence-and-timing-boundaries) +for the existing experiments and their limits. + +## Source map + +| Source | Responsibility | +| --- | --- | +| [`core/bulk.py`](../src/cuphoton/core/bulk.py) | Work items, shard assignment and completion-record checks | +| [`core/execution.py`](../src/cuphoton/core/execution.py) | Component worker contract, item execution and round validation | +| [`core/executors.py`](../src/cuphoton/core/executors.py) | Select Dragon or MPI | +| [`xscan/pipeline_executor.py`](../src/cuphoton/xscan/pipeline_executor.py) | Adapt the imaging pipeline to the shared executors | +| [`xscan/device_pipeline.py`](../src/cuphoton/xscan/device_pipeline.py) | Run the imaging stages and manage device-buffer lifetimes | diff --git a/docs/dragon-performance.md b/docs/dragon-performance.md new file mode 100644 index 00000000..44b8b48d --- /dev/null +++ b/docs/dragon-performance.md @@ -0,0 +1,212 @@ +# Dragon transport and performance + +For process placement, messages and GPU ownership, read +[How cuPhoton uses Dragon](dragon.md). For launch commands and shared-path +setup, use the [distributed launch guide](distributed-execution.md). + +## Select the application and overlay transports + +Dragon has separate application and infrastructure overlay transports. +`-t tcp -o tcp` selects its Python TCP transport for both. With a Dragon 0.14.2 +installation that includes native HSTA, select native HSTA for application +traffic and retain Python TCP for the overlay: + +```bash +DRAGON_HSTA_FORCE_BACKEND=tcp \ + .venv/bin/dragon -m -N 2 -w slurm -t hsta -o tcp \ + examples/xpois/dragon_batch.py \ + --backend cupy \ + --manifest /shared/manifests/fixed-32.yaml \ + --output-dir /shared/results/xpois-hsta \ + --name fixed-32-hsta16 \ + --max-workers 16 \ + --worker-timeout-sec 3600 +``` + +As in the xPois launch example, these paths must be accessible to the allocated +nodes. The manifest needs enough image pairs for the selected workers. Adapt +the node and worker counts to the allocation. + +`DRAGON_HSTA_FORCE_BACKEND=tcp` selects HSTA's native TCP implementation. This +configuration uses TCP rather than UCX or RDMA. Inspect the launcher and +transport logs to confirm HSTA started on each node and exited successfully; +record the Dragon version, arguments and environment with the run. INFO logs +can confirm native HSTA startup without providing network byte counters. + +HSTA TCP is a useful candidate for workloads with frequent small messages. +Qualify it on the target installation with representative numerical work, +output validation and cleanup checks. The Python TCP launch in the xPois guide +remains an explicit alternative. Transport availability and the fastest queue +layout depend on the Dragon build and workload. + +## Place command queues with their consumers + +The repeated xPois and shared Dragon executors use a READY/command/result +protocol. Each worker sends READY after initialization, then waits on its own +command queue. The coordinator waits for all READY messages before releasing +a round. Custom coordinators with the same protocol need to consider queue +placement. + +In Dragon 0.14.2's Python TCP transport, remote receives, sends and polls share +an executor. A receive waiting on an empty remote queue can occupy a thread +until a message arrives or its timeout expires. If command queues are all on +the coordinator's node, enough idle remote workers can consume the threads +needed to deliver the remaining READY messages. Both sides then wait. + +For this pattern with Python TCP, place each command queue on its consumer's +host. Keep the result queue on its consumer, the coordinator: + +```python +from dragon.infrastructure.policy import Policy +from dragon.native.machine import Node +from dragon.native.queue import Queue + +# worker_node_id comes from the allocation's discovered Dragon node IDs. +worker_host = Node(worker_node_id).hostname +worker_policy = Policy( + placement=Policy.Placement.HOST_NAME, + host_name=worker_host, +) +command_queue = Queue(maxsize=1, policy=worker_policy) +``` + +Use the hostname reported by Dragon's node discovery for both the queue and +worker placement. This makes the worker's idle receive local and the +coordinator's command send remote. It removes the central population of idle +receive waits while retaining command traffic and result collection. + +`DRAGON_TRANSPORT_TCP_MAX_THREADS` can increase the Python TCP executor ceiling. +It allowed the centralized layout to make progress in testing, but preserved +its idle remote receives and higher control latency. Native HSTA has a different +progress implementation; the Python TCP thread setting still applies to a +Python TCP overlay. Consumer-local queues are not universally faster under HSTA. + +The repeated-round and shared executors use consumer-local +command queues. The [single-pass xPois path](../src/cuphoton/xpois/dragon.py) +uses only a result queue and needs no command-queue relocation. Neither path +automatically changes the transport or raises the Python TCP thread ceiling; +those remain launcher settings. + +The shared Dragon executor publishes terminal success only after worker +shutdown and lifecycle audits. Per-round success does not establish a +successful complete run. Silent worker exits and shutdown failures fail the +run. After a reported round failure, healthy peers can finish and retain their +artifacts. Check the root terminal summary before consuming a benchmark result. + +## Evidence and timing boundaries + +A Dragon 0.14.2 control experiment used two nodes, with the coordinator on one +and 64 CPU worker processes on the other. Each worker returned a 1024-byte +payload per round; no GPU work or timed receipt-file writes occurred. Each +successful invocation had 12 rounds. Two invocations per configuration reversed +the queue-layout order. The table gives the median of rounds 2–12 for each +invocation; the first round was retained separately. + +| Application transport | Command queue host | Python TCP ceiling | Later-round medians | +|---|---|---:|---:| +| Python TCP | Coordinator | 1024 | 47.235 / 45.307 ms | +| Python TCP | Consumer | 1024 | 33.604 / 28.123 ms | +| Native HSTA TCP | Coordinator | 32 for overlay | 2.371 / 2.545 ms | +| Native HSTA TCP | Consumer | 32 for overlay | 3.027 / 3.093 ms | + +A separate invocation with coordinator queues and a 32-thread Python TCP +ceiling stalled after 32 of 64 READY messages. Consumer queues completed all +12 rounds at the same ceiling. HSTA's first rounds took 5.6–6.3 ms, compared +with the 2.4–3.1 ms later medians above. These are whole control-round timings, +not network-only measurements or predicted application speedups. + +A separate persistent-worker imaging harness compared MPI and both queue +layouts under Python TCP and HSTA TCP on eight GB200 GPUs. It processed +16 image-pair occurrences per round from two base pairs, for three rounds per +configuration. All 240 measured scientific outputs agreed. Batch medians +ranged from 7.515 to 7.719 seconds: the large control-only improvement did not +produce a comparable whole-pipeline gain. These results do not qualify a +512-GPU speedup or establish the best layout at larger node counts. This was +a separate harness experiment, with one invocation per configuration. + +Later two-node/eight-GPU product checks covered xPois, standalone xFit and +xScan, and the combined pipeline under Dragon and MPI. They established +numerical parity, persistent identities and cleanup for the tested revisions. +They predate later lifecycle, loader and input-ownership fixes and describe +the tested revisions. Revalidate the installed release before using those +measurements to characterize current executor behavior. + +### 256-GPU follow-up + +A later instrumented harness compared MPI and Dragon on 256 GPUs across +64 nodes. Fixed batches contained 512 image-pair occurrences (two per GPU); +weak batches contained 4,096 (16 per GPU). All Dragon treatments used the +1024-thread Python TCP ceiling for the applicable application or overlay +transport. The table shows the range of +batch duration minus longest worker duration across all three rounds of each +accepted Dragon invocation, including the first command round: + +| Dragon treatment | Fixed batch remainder | Weak batch remainder | +|---|---:|---:| +| Python TCP, coordinator queues | 209.590–357.407 ms | 361.694–462.434 ms | +| Python TCP, consumer queues | 24.325–32.729 ms | 39.762–45.234 ms | +| Native HSTA TCP, coordinator queues | 8.126–8.629 ms | 7.863–9.451 ms | + +The fixed-batch consumer results combine two accepted launches; each other +table cell comes from one. The complete MPI/Dragon campaign retained 10 accepted +launches, with 30 measured rounds and 58,368 validated image-pair outputs, +alongside two startup failures and four unrun launches. Consumer-local queues +and HSTA reduced the observed coordination remainder, which also includes +dispatch, serialization and collection. It is not a network-only timer. + +The coordinator-queue control also lacked the historical multi-second spike. +Allocation, scheduler segmentation and logging differed from the earlier +campaign, so the spike's cause remains unresolved. There was no 512-GPU +retest. These measurements exercised the retained harness, not the latest +product PR revisions; they do not establish large-scale performance or launch +reliability for those revisions. + +### Compare timing boundaries + +For a new comparison, hold the corpus, item count, numerical configuration, +worker CPU budget and runtime versions constant. Rotate run order and retain +the first command round as well as subsequent rounds. Record separately: + +- Launcher-to-exit wall time, including startup, warmup and shutdown. +- Time from harness or executor entry until all workers report READY. +- Whole-batch time from release through collection of worker completions. +- Worker execution time, with the treatment of output and receipt writes. +- Coordinator artifact audits, scientific finalization and shutdown. + +### Executor timing fields + +In persistent-round reports, `batch_wall_sec` stops at completion collection; +coordinator artifact audits and scientific finalization follow it. The shared +executor records component merging separately as `finalization_sec`. Keep +readiness and shutdown outside this batch interval, and measure external +launcher-to-exit time independently. MPI finalization and process exit are +outside the executor's reported coordinator time. + +Requested warmup rounds exercise real dispatch, work and collection. Their +outputs and timings remain in the report. Only measured rounds enter timing +statistics; failed warmup rounds, measured rounds or cleanup invalidate the +aggregate. A numerical warmup performed before READY in a separate harness +does not necessarily warm the first command round. Match these policies before +comparing results. + +### Interpreting comparisons + +Subtracting the longest worker duration from the whole-batch duration combines +start skew, synchronization, receipt writes, serialization and collection. It +cannot isolate network time. Validate complete output identities and scientific +results, worker exits and runtime cleanup alongside timings. The control probe +also observed an external Dragon launcher exit of zero after an application +failure; inspect the application's terminal result as well as the launcher. + +[The pipeline/stage benchmark](components/pipeline-stage-benchmark.md) +compares a resident device pipeline with fresh processes and intermediate +files on one GPU. It measures workflow reuse and process/file costs, retaining +raw timings and separately reporting the cost of additional hash verification. +It does not compare Dragon transports or reproduce the stock component CLI +chain. + +Free-threaded Python does not free an executor thread blocked in a native +receive: that receive already releases the GIL. A separate serialization or +dispatch profile is needed to establish whether Python execution is a remaining +bottleneck. The experiments above used conventional CPython 3.12; they do not +qualify a free-threaded Dragon runtime. diff --git a/docs/dragon.md b/docs/dragon.md index d0c18295..2736613e 100644 --- a/docs/dragon.md +++ b/docs/dragon.md @@ -1,254 +1,306 @@ -# Dragon transport and coordination - -Dragon places and manages cuPhoton workers; the selected numerical backend -runs each image pair inside its worker. Transport selection belongs to the -Dragon launcher. It does not change cuPhoton's numerical backend or fit options. - -Start with [distributed execution](distributed-execution.md) for environment -setup, a pipeline manifest, Slurm allocations and representative host files. - -## Execution models - -The [xPois batch executor](components/xpois.md#launch-with-dragon) -assigns a complete shard to each worker at launch. Workers process their -assigned items and send one terminal result to the coordinator in single-pass -mode. Repeated rounds and component commands retain workers as follows: - -| Path | Coordination and retained state | -|---|---| -| Single-pass xPois | One assigned shard and one terminal result per worker; no per-round command queue. | -| [Repeated xPois rounds](components/xpois.md#repeat-a-batch-in-persistent-workers) | `--warmup-rounds` and `--measure-rounds` reuse workers and CUDA contexts. Each worker has a command queue on its own host and repeats its assigned shard. | -| [Shared executors](../src/cuphoton/core/executors.py) and [component commands](components/xscan.md#distributed-inference) | xFit, xScan and `xscan run-pipeline` use the shared Dragon/MPI lifecycle. Dragon uses consumer-local command queues for both a single pass and repeated rounds; component workers retain their input/model context. | - -### Persistent behavior - -Persistence does not remove all per-item work. xPois still reads, transfers and -writes ordinary scientific artifacts each round. Standalone xFit retains its -loaded host input; xScan retains metadata, its model and loader across tasks. -Distributed xScan defaults to `--num-workers 0`. Explicitly requested loader -processes start before READY and persist across tasks and rounds. The combined -pipeline keeps intermediate arrays inside each GPU worker while it -processes an image pair. Each GPU handles complete items; these paths do not -split one image across GPUs. - -The shared executor accepts `--workers-per-gpu` to place several independent -workers on each GPU; its default is one. `--max-workers` remains a total -process cap. See [GPU sharing](components/xscan.md#share-a-gpu-between-image-pairs) -for CLI and Python examples, local process/thread modes, and external MPS -connection checks. These sharing and MPS options apply to `xfit fit-dipoles`, -`xscan infer-real-bogus`, and `xscan run-pipeline` with `--executor dragon`. -The standalone XPOIS batch API retains its existing placement behavior. - -### Launch descriptors - -Dragon workers load immutable, hash-checked launch descriptors from the shared -filesystem. Launch arguments carry descriptor references rather than the full -manifest, with a 96 KiB size guard. This bounds process-launch payloads; queue -placement addresses a separate coordination cost. - -## Select the application and overlay transports - -Dragon has separate application and infrastructure overlay transports. -`-t tcp -o tcp` selects its Python TCP transport for both. With a Dragon 0.14.2 -installation that includes native HSTA, select native HSTA for application -traffic and retain Python TCP for the overlay: - -```bash -DRAGON_HSTA_FORCE_BACKEND=tcp \ - .venv/bin/dragon -m -N 2 -w slurm -t hsta -o tcp \ - examples/xpois/dragon_batch.py \ - --backend cupy \ - --manifest /shared/manifests/fixed-32.yaml \ - --output-dir /shared/results/xpois-hsta \ - --name fixed-32-hsta16 \ - --max-workers 16 \ - --worker-timeout-sec 3600 +# How cuPhoton uses Dragon + +cuPhoton uses Dragon to place Python workers on GPU nodes and collect their +completion messages. Each worker is bound to one GPU and runs complete work +items. By default, cuPhoton places one worker on each GPU. Optional sharing +lets several independent workers use the same GPU. The coordinator assigns +work, checks results and stops the workers. + +Dragon carries control messages and result metadata. In the combined imaging +pipeline, each worker runs xPois, xFit and xScan on its assigned image pairs. +Intermediate image arrays stay on that worker's GPU. + +The [distributed architecture](distributed.md) describes the common work and +data model. The [launch guide](distributed-execution.md) provides runnable +examples. [Transport and performance](dragon-performance.md) covers TCP, +HSTA, queue placement experiments and timing limits. + +## Processes and queues + +The Dragon launcher starts the cuPhoton coordinator within the Dragon +runtime. The coordinator uses Dragon's native API to discover nodes, select +GPUs and create workers. It creates one `ProcessGroup` containing one +`ProcessTemplate` per worker. + +The shared executor uses one command queue per worker and one common result +queue. The coordinator creates each command queue with a placement policy +for that worker's host. The result queue resides with the coordinator. + +```mermaid +flowchart TB + L["Dragon launcher"] --> C + subgraph A["Node A"] + C["cuPhoton coordinator"] + Q["Common result queue"] + QA["Command queue A"] + WA["Python worker A: GPU 0"] + QA -->|"Local receive"| WA + Q -->|"Local receive"| C + end + subgraph B["Node B"] + QB["Command queue B"] + WB["Python worker B: GPU 0"] + QB -->|"Local receive"| WB + end + C --> QA + C -->|"Remote send"| QB + WA --> Q + WB -->|"Remote send"| Q ``` -As in the xPois launch example, these paths must be accessible to the allocated -nodes. The manifest needs enough image pairs for the selected workers. Adapt -the node and worker counts to the allocation. - -`DRAGON_HSTA_FORCE_BACKEND=tcp` selects HSTA's native TCP implementation. This -configuration uses TCP rather than UCX or RDMA. Inspect the launcher and -transport logs to confirm HSTA started on each node and exited successfully; -record the Dragon version, arguments and environment with the run. INFO logs -can confirm native HSTA startup without providing network byte counters. - -HSTA TCP is a useful candidate for workloads with frequent small messages. -Qualify it on the target installation with representative numerical work, -output validation and cleanup checks. The Python TCP launch in the xPois guide -remains an explicit alternative. Transport availability and the fastest queue -layout depend on the Dragon build and workload. - -## Place command queues with their consumers - -The repeated xPois and shared Dragon executors use a READY/command/result -protocol. Each worker sends READY after initialization, then waits on its own -command queue. The coordinator waits for all READY messages before releasing -a round. Custom coordinators with the same protocol need to consider queue -placement. - -In Dragon 0.14.2's Python TCP transport, remote receives, sends and polls share -an executor. A receive waiting on an empty remote queue can occupy a thread -until a message arrives or its timeout expires. If command queues are all on -the coordinator's node, enough idle remote workers can consume the threads -needed to deliver the remaining READY messages. Both sides then wait. - -For this pattern with Python TCP, place each command queue on its consumer's -host. Keep the result queue on its consumer, the coordinator: - -```python -from dragon.infrastructure.policy import Policy -from dragon.native.machine import Node -from dragon.native.queue import Queue - -# worker_node_id comes from the allocation's discovered Dragon node IDs. -worker_host = Node(worker_node_id).hostname -worker_policy = Policy( - placement=Policy.Placement.HOST_NAME, - host_name=worker_host, -) -command_queue = Queue(maxsize=1, policy=worker_policy) +This example shows the default arrangement, with one worker per GPU. The coordinator is a separate process +and needs no assigned GPU. Dragon's own runtime services are omitted from +the diagram. Each node can host several workers. Each worker has its own +command queue and one assigned GPU. + +The queues contain Python metadata. They do not contain image pixels, model +weights or CUDA buffers. Workers read input and model files directly from +shared storage, then write their outputs there. A result message includes +status, item identities, placement and timing. Its size can grow with the +number of assigned items. + +## Select hosts and bind GPUs + +cuPhoton reads the node list through `System` and each node's GPU IDs through +`Node.gpus`. It uses the reported IDs, including nonconsecutive IDs. Selection +rotates across hosts before taking a second GPU from each host. It uses +distinct GPUs before assigning another worker to a GPU. + +The shared executor limits workers to the item count and the available GPU +count multiplied by `--workers-per-gpu`. The default is one worker per GPU. +`--max-workers` caps the total process count. cuPhoton assigns each worker +a fixed shard of items. A Dragon `Policy` specifies the hostname and one GPU affinity +for that process. + +Before creating the component worker, cuPhoton checks the actual hostname and +`CUDA_VISIBLE_DEVICES`. The worker must see exactly the requested GPU. +The check rejects prior imports of CuPy, Numba CUDA or cuTile, or an +initialized Torch CUDA runtime. The component then initializes its numerical libraries and uses local device 0. +The worker reports its physical GPU identity. Workers assigned to the same +GPU must agree on that identity. Workers assigned to different GPUs must +report different identities. + +This ordering matters because a CUDA library can retain the device selection +from its first initialization. Changing the visibility environment afterward +does not move an existing CUDA context. + +## Share a GPU between workers + +`--workers-per-gpu N` permits up to N worker processes on each GPU. For +example, four GPUs with two workers per GPU provide eight worker slots. +Item count and `--max-workers` can reduce the actual worker count. + +Each worker processes complete items and retains its own component state. The combined pipeline therefore +loads a separate model for each worker. More workers can increase memory +use and contention. Measure batch time and memory use for the intended inputs +before choosing a worker count. + +Each process still has a command queue and sends results to the common +result queue. More workers mean more queues, launch descriptors and completion +messages. The shared lifecycle below applies to every worker. + +### Connect workers to MPS + +The worker count and MPS connection are separate choices. Multiple processes +can share a GPU under ordinary CUDA scheduling. An MPS service allows CUDA +work from different processes to overlap on the device. + +`--mps-pipe-directory PATH` requires every worker to connect to an existing +MPS v2 service. The launcher or administrator starts that service and owns +its resource limits and shutdown. cuPhoton does not start or stop MPS or +change GPU compute mode. + +```mermaid +flowchart TB + subgraph N["One GPU host"] + M["External MPS service"] + W0["Dragon worker 0
Own model and item shard"] + W1["Dragon worker 1
Own model and item shard"] + G["One physical GPU"] + M -.-|"Client PID check"| W0 + M -.-|"Client PID check"| W1 + W0 -->|"CUDA work"| G + W1 -->|"CUDA work"| G + end ``` -Use the hostname reported by Dragon's node discovery for both the queue and -worker placement. This makes the worker's idle receive local and the -coordinator's command send remote. It removes the central population of idle -receive waits while retaining command traffic and result collection. - -`DRAGON_TRANSPORT_TCP_MAX_THREADS` can increase the Python TCP executor ceiling. -It allowed the centralized layout to make progress in testing, but preserved -its idle remote receives and higher control latency. Native HSTA has a different -progress implementation; the Python TCP thread setting still applies to a -Python TCP overlay. Consumer-local queues are not universally faster under HSTA. - -The repeated-round and shared executors use consumer-local -command queues. The [single-pass xPois path](../src/cuphoton/xpois/dragon.py) -uses only a result queue and needs no command-queue relocation. Neither path -automatically changes the transport or raises the Python TCP thread ceiling; -those remain launcher settings. - -The shared Dragon executor publishes terminal success only after worker -shutdown and lifecycle audits. Per-round success does not establish a -successful complete run. Silent worker exits and shutdown failures fail the -run. After a reported round failure, healthy peers can finish and retain their -artifacts. Check the root terminal summary before consuming a benchmark result. - -## Evidence and timing boundaries - -A Dragon 0.14.2 control experiment used two nodes, with the coordinator on one -and 64 CPU worker processes on the other. Each worker returned a 1024-byte -payload per round; no GPU work or timed receipt-file writes occurred. Each -successful invocation had 12 rounds. Two invocations per configuration reversed -the queue-layout order. The table gives the median of rounds 2–12 for each -invocation; the first round was retained separately. - -| Application transport | Command queue host | Python TCP ceiling | Later-round medians | -|---|---|---:|---:| -| Python TCP | Coordinator | 1024 | 47.235 / 45.307 ms | -| Python TCP | Consumer | 1024 | 33.604 / 28.123 ms | -| Native HSTA TCP | Coordinator | 32 for overlay | 2.371 / 2.545 ms | -| Native HSTA TCP | Consumer | 32 for overlay | 3.027 / 3.093 ms | - -A separate invocation with coordinator queues and a 32-thread Python TCP -ceiling stalled after 32 of 64 READY messages. Consumer queues completed all -12 rounds at the same ceiling. HSTA's first rounds took 5.6–6.3 ms, compared -with the 2.4–3.1 ms later medians above. These are whole control-round timings, -not network-only measurements or predicted application speedups. - -A separate persistent-worker imaging harness compared MPI and both queue -layouts under Python TCP and HSTA TCP on eight GB200 GPUs. It processed -16 image-pair occurrences per round from two base pairs, for three rounds per -configuration. All 240 measured scientific outputs agreed. Batch medians -ranged from 7.515 to 7.719 seconds: the large control-only improvement did not -produce a comparable whole-pipeline gain. These results do not qualify a -512-GPU speedup or establish the best layout at larger node counts. This was -a separate harness experiment, with one invocation per configuration. - -Later two-node/eight-GPU product checks covered xPois, standalone xFit and -xScan, and the combined pipeline under Dragon and MPI. They established -numerical parity, persistent identities and cleanup for the tested revisions. -They predate later lifecycle, loader and input-ownership fixes and describe -the tested revisions. Revalidate the installed release before using those -measurements to characterize current executor behavior. - -### 256-GPU follow-up - -A later instrumented harness compared MPI and Dragon on 256 GPUs across -64 nodes. Fixed batches contained 512 image-pair occurrences (two per GPU); -weak batches contained 4,096 (16 per GPU). All Dragon treatments used the -1024-thread Python TCP ceiling for the applicable application or overlay -transport. The table shows the range of -batch duration minus longest worker duration across all three rounds of each -accepted Dragon invocation, including the first command round: - -| Dragon treatment | Fixed batch remainder | Weak batch remainder | -|---|---:|---:| -| Python TCP, coordinator queues | 209.590–357.407 ms | 361.694–462.434 ms | -| Python TCP, consumer queues | 24.325–32.729 ms | 39.762–45.234 ms | -| Native HSTA TCP, coordinator queues | 8.126–8.629 ms | 7.863–9.451 ms | - -The fixed-batch consumer results combine two accepted launches; each other -table cell comes from one. The complete MPI/Dragon campaign retained 10 accepted -launches, with 30 measured rounds and 58,368 validated image-pair outputs, -alongside two startup failures and four unrun launches. Consumer-local queues -and HSTA reduced the observed coordination remainder, which also includes -dispatch, serialization and collection. It is not a network-only timer. - -The coordinator-queue control also lacked the historical multi-second spike. -Allocation, scheduler segmentation and logging differed from the earlier -campaign, so the spike's cause remains unresolved. There was no 512-GPU -retest. These measurements exercised the retained harness, not the latest -product PR revisions; they do not establish large-scale performance or launch -reliability for those revisions. - -### Compare timing boundaries - -For a new comparison, hold the corpus, item count, numerical configuration, -worker CPU budget and runtime versions constant. Rotate run order and retain -the first command round as well as subsequent rounds. Record separately: - -- Launcher-to-exit wall time, including startup, warmup and shutdown. -- Time from harness or executor entry until all workers report READY. -- Whole-batch time from release through collection of worker completions. -- Worker execution time, with the treatment of output and receipt writes. -- Coordinator artifact audits, scientific finalization and shutdown. - -### Executor timing fields - -In persistent-round reports, `batch_wall_sec` stops at completion collection; -coordinator artifact audits and scientific finalization follow it. The shared -executor records component merging separately as `finalization_sec`. Keep -readiness and shutdown outside this batch interval, and measure external -launcher-to-exit time independently. MPI finalization and process exit are -outside the executor's reported coordinator time. - -Requested warmup rounds exercise real dispatch, work and collection. Their -outputs and timings remain in the report. Only measured rounds enter timing -statistics; failed warmup rounds, measured rounds or cleanup invalidate the -aggregate. A numerical warmup performed before READY in a separate harness -does not necessarily warm the first command round. Match these policies before -comparing results. - -### Interpreting comparisons - -Subtracting the longest worker duration from the whole-batch duration combines -start skew, synchronization, receipt writes, serialization and collection. It -cannot isolate network time. Validate complete output identities and scientific -results, worker exits and runtime cleanup alongside timings. The control probe -also observed an external Dragon launcher exit of zero after an application -failure; inspect the application's terminal result as well as the launcher. - -[The pipeline/stage benchmark](components/pipeline-stage-benchmark.md) -compares a resident device pipeline with fresh processes and intermediate -files on one GPU. It measures workflow reuse and process/file costs, retaining -raw timings and separately reporting the cost of additional hash verification. -It does not compare Dragon transports or reproduce the stock component CLI -chain. - -Free-threaded Python does not free an executor thread blocked in a native -receive: that receive already releases the GIL. A separate serialization or -dispatch profile is needed to establish whether Python execution is a remaining -bottleneck. The experiments above used conventional CPython 3.12; they do not -qualify a free-threaded Dragon runtime. +The diagram shows two workers sharing a GPU with MPS. Each retains the +Dragon queues shown in the earlier diagram. MPS connection checks are +separate from Dragon's round commands and results. + +Before creating the component worker, cuPhoton checks Dragon's initial GPU +binding. It then sets `CUDA_MPS_PIPE_DIRECTORY`, resolves the device's UUID +with `nvidia-smi`, and replaces `CUDA_VISIBLE_DEVICES` with that UUID. All +three steps happen before CUDA initialization. UUID binding avoids changes +in device numbering under MPS. Each process still uses local device 0. + +After CUDA initialization, cuPhoton queries the MPS control daemon for its +servers and their clients. The queries have a shared 10-second deadline. +The worker PID must appear in a returned client list. + +The worker reports readiness only after that check succeeds. Its startup record includes the GPU UUID and the MPS +client and server PIDs. A missing service or failed connection check fails +startup. + +On multiple hosts, use the same absolute, node-local pipe path on every +worker host. The MPS service and its clients must use the same PID namespace +for the connection check. Without the explicit pipe option, cuPhoton leaves +inherited MPS settings in effect and does not verify an MPS connection. +Omitting the option does not prove that MPS is disabled. + +Sharing and MPS options apply to `xfit fit-dipoles`, `xscan infer-real-bogus` +and `xscan run-pipeline` with `--executor dragon`. Standalone xPois and the +Python `run_dragon_device_pipeline()` API retain one worker per GPU. MPI +also retains one rank per physical GPU and rejects these sharing options. +The [GPU sharing guide](components/xscan.md#share-a-gpu-between-image-pairs) +contains commands and describes the separate local process and thread modes. + +## Load launch descriptors from shared files + +Here, a *launch descriptor* is a JSON file with a worker's assigned items, +options and placement. It is not an operating-system file descriptor. + +The coordinator writes one descriptor per worker under the run's `launch/` +directory. It passes the path, expected SHA-256, small validation context and +queue handles as process arguments. The worker checks the descriptor before +it imports and creates the component worker. + +This keeps a large manifest out of each process-launch request. +cuPhoton rejects serialized launch arguments larger than 96 KiB. That guard +applies to the arguments, not to all queue messages or the descriptor file. +The shared filesystem must remain accessible during worker startup. + +## Shared worker lifecycle + +xFit, xScan inference and the `xscan run-pipeline` command use the shared +executor in `cuphoton.core.dragon`. They use this protocol for one round as +well as repeated rounds: + +```mermaid +sequenceDiagram + participant C as Coordinator + participant W as Each GPU worker + participant F as Shared filesystem + C->>F: Write launch descriptors + C->>W: Start native process with descriptor reference + W->>F: Read and validate descriptor + W->>W: Check GPU binding
set MPS pipe and UUID if requested
create component worker + opt Explicit MPS pipe selected + W->>W: Verify MPS client connection + end + W-->>C: ready: identity and startup status + C->>C: Validate all workers + loop Each requested round + C->>W: Round identity on worker command queue + W->>W: Run all items in assigned shard + W->>F: Write outputs and completion records + W-->>C: round: completion metadata + C->>F: Read and validate records
merge outputs + end + C->>W: close + W->>W: Close component resources + W-->>C: closed: cleanup status + C->>C: Join workers
check exits
close group and queues + C->>F: Publish final summary.json +``` + +The diagram shows a successful run. The coordinator waits for every `ready` +message before it releases the first round. Each round command identifies +the run and round; it does not send a new shard. The same component object +processes every requested round. + +Each command queue has capacity one. The result queue has capacity +for twice the worker count. After the final round, workers wait for `close` +before they send `closed`. This lets the coordinator finish collecting and +checking round results before normal shutdown starts. + +## Why queue placement matters + +An idle worker blocks in a receive on its own command queue. Placing +that queue on the worker's host keeps the wait local. The coordinator sends +a small message to that host when a round can start. + +With Dragon 0.14.2's Python TCP transport, many idle receives on remote queues +can consume transport executor threads. If every command queue resides on +the coordinator's node, those waits can delay other control messages. +cuPhoton's repeated-round and shared paths place command queues with their +consumers to avoid that layout. + +The Dragon launcher selects application and overlay transports. cuPhoton does +not select a transport or change its thread limits. Native HSTA uses a +different progress implementation. The +[transport guide](dragon-performance.md#place-command-queues-with-their-consumers) +explains the observed behavior and the measurements behind this choice. + +## xPois uses a separate executor + +The xPois batch command predates the shared executor and retains a different +protocol: + +| Entry point | Worker protocol | +| --- | --- | +| `xpois fit-batch`, ordinary run | Process one assigned shard, send one result and exit. No command queue or readiness handshake. | +| `xpois fit-batch`, repeated rounds | Report readiness, wait for round commands and repeat the same shard. Exit after the final round, without the shared `close`/`closed` handshake. | +| `xscan run-pipeline` command | Use the shared lifecycle shown above. | +| Python `run_dragon_device_pipeline()` | Use the single-pass xPois coordinator. Retain one device context across the items in a shard. | + +Ordinary xPois gives its result queue one slot per worker. The coordinator +joins the workers before draining those results. Repeated xPois adds command +queues on the workers' hosts. Both paths use native Dragon processes and +fixed assignments. + +The word *persistent* depends on the entry point. In a single-pass path, +a context persists across the items of one shard. In repeated rounds, it +persists across complete rounds. + +## Local xScan loader processes + +Distributed xScan inference defaults to `--num-workers 0`, so the GPU worker +also loads its batches. A positive value requests local data-loader child +processes for each GPU worker. They prepare batches; the parent runs the +model on its GPU. + +The shared Dragon executor sets `DRAGON_PATCH_MP=""` in each native worker's +environment. Python's multiprocessing module then stays unpatched in that +worker. Spawned loader children receive ordinary multiprocessing queues. +The xScan CUDA loader defaults to the `spawn` start method. + +The outer GPU workers still use native Dragon processes and queues. Loader +children start before the GPU worker reports `ready`, persist across tasks +and rounds, and stop when the component worker closes. + +## Failures and shutdown + +cuPhoton creates the process group with restart disabled. cuPhoton does not move failed +items to healthy workers. An item exception produces a failed record, and +the shared worker continues through its remaining items. A failed round +prevents the next round from starting. + +The coordinator checks received messages against the run, worker and round +identities. It also checks files on disk, physical GPU assignments and process +exit status. Missing results, unexpected exits and failed cleanup make the +run fail. + +On failure, the coordinator attempts to stop the group and close its queues. +A forced process termination can prevent Python cleanup or a final `closed` +message. Retained records describe completed work and observed errors. They do not +show that every worker completed cleanup. + +For a successful shared run, the coordinator publishes terminal success only +after worker shutdown and lifecycle checks. Inspect the root `summary.json` +as well as the launcher exit status. Use the timeout and cleanup guidance in +[the launch guide](distributed-execution.md#inspect-the-run). + +## Source map + +| Source | Responsibility | +| --- | --- | +| [`core/dragon.py`](../src/cuphoton/core/dragon.py) | Discovery, placement, queues, shared worker protocol and shutdown | +| [`core/execution.py`](../src/cuphoton/core/execution.py) | Component construction contract, item records and validation | +| [`core/mps.py`](../src/cuphoton/core/mps.py) | Verify client connections to an externally managed MPS v2 service | +| [`xpois/dragon.py`](../src/cuphoton/xpois/dragon.py) | xPois single-pass and repeated-round protocols | +| [`xscan/dragon_pipeline.py`](../src/cuphoton/xscan/dragon_pipeline.py) | Python device-pipeline API using the single-pass coordinator | +| [`xscan/executor.py`](../src/cuphoton/xscan/executor.py) | xScan model and local loader lifetime | + +The upstream [Dragon native API reference](https://dragonhpc.github.io/dragon/doc/_build/html/ref/native/index.html) +describes the process, queue and placement primitives used here. diff --git a/docs/mpi.md b/docs/mpi.md new file mode 100644 index 00000000..18a9c23c --- /dev/null +++ b/docs/mpi.md @@ -0,0 +1,127 @@ +# How cuPhoton uses MPI + +cuPhoton uses MPI to run the same independent GPU work described in the +[distributed architecture](distributed.md). The MPI launcher starts all +processes. Each process has a rank and uses one assigned GPU. Rank 0 also coordinates +the run while computing its own assigned shard. + +The [launch guide](distributed-execution.md) provides Open MPI commands for +Slurm and SSH. [How cuPhoton uses Dragon](dragon.md) describes the alternative +process and queue model. MPI retains one rank per physical GPU and rejects +`--workers-per-gpu` and `--mps-pipe-directory`. + +## Bind each rank before CUDA starts + +For Open MPI, `cuphoton-openmpi-rank-exec` runs before Python starts. It selects +one entry from the allocation's `CUDA_VISIBLE_DEVICES` list using the rank's +position on its node. It preserves the original list in +`CUPHOTON_ALLOCATED_CUDA_VISIBLE_DEVICES`, then executes the application. + +The shared executor requires one visible GPU before it imports mpi4py. +This protects against MPI installations that initialize CUDA during import. +The numerical code uses device 0 within each rank's restricted visibility. + +The wrapper expects the full per-node GPU list in local-rank order. It +rejects a local rank outside that list. When the scheduler already gives each task one visible GPU, launch without +the wrapper. Other MPI implementations +need their own launcher or scheduler binding. + +After MPI starts, cuPhoton compares the launcher's rank variables with the +MPI communicator and collects physical GPU identities. It rejects duplicate GPU assignments. The shared executor also +rejects more ranks than work items, so each rank has work. + +## Shared executor lifecycle + +xFit, xScan inference and `xscan run-pipeline` use `cuphoton.core.mpi`. +Rank 0 creates the run directory and coordinates validation. All ranks create +one component worker and retain it through the requested rounds. + +```mermaid +sequenceDiagram + participant L as MPI launcher + participant Z as Rank 0 + participant R as Other GPU ranks + L->>Z: Start with one GPU visible + L->>R: Start with one GPU visible + Z->>R: Agree workload and shared run directory + Z->>Z: Create component worker + R->>R: Create component worker + R-->>Z: Gather startup status and GPU identity + Z->>R: Broadcast readiness decision + loop Each requested round + Z->>R: Broadcast round identity + par Rank 0 computes too + Z->>Z: Run fixed shard
write files + and Other ranks compute + R->>R: Run fixed shard
write files + end + R-->>Z: Gather completion metadata + Z->>Z: Validate files and merge outputs + Z->>R: Broadcast round decision + end + Z->>Z: Close component worker + R->>R: Close component worker + R-->>Z: Gather cleanup status + Z->>Z: Write final summary.json + Z->>R: Broadcast final status + Note over L,R: MPI finalization and process exit follow +``` + +These exchanges use collectives. Every rank, including rank 0, participates +in each `gather` and `bcast` call. They are not independent command +queues. Rank 0 executes its shard before it completes result collection. + +The shared executors exchange Python metadata through MPI. Workers read +inputs and write scientific outputs through the shared filesystem. A +readiness token checks that all ranks see the same run directory. The +coordinator compares completion messages with the records on disk before +it accepts a round. + +No imaging stage requires an MPI reduction of pixels or GPU arrays. Each +worker completes its own items. Scientific finalization joins component +outputs after the workers have written them. + +## What happens after a failure + +Ordinary item exceptions become failed records. Each rank continues through +its remaining assigned items, then participates in completion collection. +A failed round stops subsequent rounds. Errors during component cleanup +also make the final run fail. + +A dead or hung rank can prevent a collective from completing. The +`--rank-setup-timeout-sec` value bounds shared-filesystem waits only. It does +not bound MPI collectives or stop rank processes. +The MPI launcher and scheduler must enforce process-failure and job-timeout +policies. cuPhoton does not replace failed ranks or redistribute their work. + +A successful final summary covers the application checks and component +cleanup. MPI finalization and process exit happen afterward. Check the +launcher result as well as the summary when recording a successful run. + +## xPois collective and file modes + +The xPois `fit-batch` command has its own MPI executor. It assigns whole +image pairs to ranks, with rank 0 participating in the numerical work. +Its collective mode supports persistent warmup and measured rounds. + +The xPois file mode uses the launcher's rank environment without importing +mpi4py. Ranks publish preflight and completion information through shared +files. Rank 0 validates and aggregates those files. This mode requires +explicit run and attempt identities and does not support repeated rounds. + +File-mode peers return after publishing their completion records. Rank 0's +aggregation timeout does not terminate peers that are still running. See +[the xPois MPI guide](components/xpois.md#launch-with-mpi) +for commands and recovery behavior. + +## Source map + +| Source | Responsibility | +| --- | --- | +| [`core/mpi.py`](../src/cuphoton/core/mpi.py) | Shared collective lifecycle, readiness, round decisions and cleanup | +| [`core/_mpi_runtime.py`](../src/cuphoton/core/_mpi_runtime.py) | Rank discovery and GPU binding checks | +| [`cuphoton-openmpi-rank-exec`](../scripts/cuphoton-openmpi-rank-exec) | Set GPU visibility before Python and MPI startup | +| [`xpois/mpi.py`](../src/cuphoton/xpois/mpi.py) | xPois collective and file modes | + +The [common timing description](distributed.md#timing-boundaries) separates +batch execution from validation, startup and process teardown. From ad317a3233e478f2b64afe2f15c99d9759ce5cec Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Fri, 9 Oct 2026 12:13:21 -0700 Subject: [PATCH 2/2] Clarify MPI startup binding and error consensus Signed-off-by: Trent Nelson --- docs/mpi.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/mpi.md b/docs/mpi.md index 18a9c23c..1135551c 100644 --- a/docs/mpi.md +++ b/docs/mpi.md @@ -17,8 +17,10 @@ one entry from the allocation's `CUDA_VISIBLE_DEVICES` list using the rank's position on its node. It preserves the original list in `CUPHOTON_ALLOCATED_CUDA_VISIBLE_DEVICES`, then executes the application. -The shared executor requires one visible GPU before it imports mpi4py. -This protects against MPI installations that initialize CUDA during import. +The launcher wrapper or scheduler must restrict each rank to one visible GPU +before Python starts. This protects against MPI installations that initialize +CUDA during import. The shared executor checks GPU visibility, then imports +mpi4py even if that check failed so all ranks can agree on startup errors. The numerical code uses device 0 within each rank's restricted visibility. The wrapper expects the full per-node GPU list in local-rank order. It