[rhc54]

The DVM half of making a survivable job's fences stop waiting for processes that have gone - a ULFM job that lost a process hung in MPI_Finalize. The PMIx half is openpmix/openpmix#4391.

A daemon whose processes have all departed

A whole-namespace fence counts every daemon hosting one of the namespace's processes, and a daemon contributes once its PMIx server has heard from every local participant. The PMIx server takes a dead client out of that count, so a daemon that loses some of its processes still contributes. One that loses all of them has nobody left to call the fence and never contributes, and its parent's rollup waited for it forever.

Each daemon now reports the termination of a process it hosts up the routing tree (PRTE_RML_TAG_FENCE_DEPARTED), and a rollup short of its count asks who it is still waiting for: a participant is awaited unless it has departed, its daemon has failed, or its subtree has reported. Up the tree rather than by broadcast, because a process that called the fence and then died has its daemon's contribution on the same link, ahead of the news of its death; a tracker records which subtrees reported, so nothing is counted twice. Only PRTE_JOB_RECOVERABLE / PRTE_JOB_CONTINUOUS jobs, nothing is spent when nothing has departed, and the fence's status is left alone - failing it would make every survivor's PMIx server discard the data the survivors contributed.

Why it went unseen: the HNP already drops an emptied node from its own copy of the job map, so in any DVM where it is every daemon's parent - up to rml_base_radix + 1 daemons, 65 by default - the hang never showed.

A daemon whose own share of the job has ended

job_teardown gives up a job's map when the daemon's processes of the job are done, while the job runs on elsewhere - and create_dmns() refused any fence over a job with no map on a daemon other than the controller, dropping the contributions of every daemon below it. A wildcard is now resolved through the job's processes, as a named rank always was. This one is not specific to failures.

Retiring and restarting clients

• A daemon now deregisters each terminated client with its PMIx server, as the HNP always has - otherwise a rank that finalized and exited stays expected until its whole job is deregistered. Only against a PMIx with PMIX_CAP_DEREG_DEPARTS (new configure check); against an older one, deregistering before the server reads the closed socket causes the very hang this fixes.
• A process restarted under its old rank is a participant again. restart_proc now reports the return up the tree (prte_grpcomm_proc_restarted()) and registers the rank with PMIx again, before the new process exists - termination deregisters it, on the HNP today, and the server refuses a rank it does not have. A recovery restart rebuilds the departed set from each daemon's own processes rather than topping it up, so a return lost with a failed daemon cannot leave a stale departure; notices carry the recovery epoch. The termination hooks sit outside the RECORDED guard, so a restarted process's next death is reported whether or not the restart resets it - that, and num_terminated, are left for whoever rewires the restart path (noted in src/mca/odls/AGENTS.md).

Testing

• test/unit/grpcomm: test_fence_departed (completion on departure, a restart, the rebuild, the epoch) and test_fence_tracker_unmapped_daemon; with each fix disabled its cases fail.
• make check: 29 pass, 0 fail.
• dockerswarm, four nodes one rank each, --prtemca rml_base_radix 2 --rtos recoverable, one rank SIGKILLed after a fence, the survivors then fencing over the whole job: on master the survivors time out with nothing completed; with this, they complete with PMIX_SUCCESS whichever rank is killed. No-death and the default-radix three-node case complete; a job that is not recoverable still aborts at its first loss. The harness's core and server suites pass.

Not covered: the group collective over a whole namespace has the same shape, but whether it may complete without a member is the caller's (PMIX_GROUP_FT_COLLECTIVE), so it needs its own policy. And a restart is unit-tested only, since nothing invokes restart_proc yet.