Skip to content

Attribute a task's work by SLURM_JOB_ID, and decline to judge threaded tasks - #63

Merged
smjenness merged 2 commits into
mainfrom
dev/doctor-multithreaded-workloads
Jul 29, 2026
Merged

Attribute a task's work by SLURM_JOB_ID, and decline to judge threaded tasks#63
smjenness merged 2 commits into
mainfrom
dev/doctor-multithreaded-workloads

Conversation

@smjenness

Copy link
Copy Markdown
Collaborator

Stacked on #62. Review that one first; the base here is dev/doctor-idle-not-starved, and I will retarget to main once it merges.

pgrep -x R decided which processes belonged to a task, which assumes the work happens in R. An rstan/cmdstanr task is an idle R wrapper plus a compiled model_<hash> binary running the chains, so the probe reported nproc=1 med_cpu=0 for a task consuming four cores. It survived only by accident, spared by the nproc < MIN_PROC startup rule. A second R process, or a moment between phases, and the doctor would have requeued a healthy Bayesian fit. Observed on a live 23-task campaign sharing this cluster.

Three changes

Membership by SLURM_JOB_ID, not by process name. Children inherit it, so the compiled binary is attributed correctly. Verified against a running cmdstan array where the model binary carries both SLURM_JOB_ID and SLURM_ARRAY_TASK_ID. Login shells and the probe's own processes have no such variable and drop out for free. The batch script wrapper is excluded by name, so nproc keeps meaning the workload and a starting task keeps its MIN_PROC grace period.

Decline to judge threaded tasks. Seeing the work is not the same as being able to judge it. The detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal and gives the 99-against-51 gap its width. A threaded binary at several hundred percent has no meaningful median, does not show the ~50% signature, and carries nothing on the node to say how many threads it asked for, so a healthy 4-thread chain and a contended 8-thread one are indistinguishable from outside. Tasks whose busiest process exceeds MULTICORE_CPU (default 150, clear of any single-threaded R process including threaded-BLAS bursts) are reported threaded and skipped. Under-flagging is the right failure here.

Report the node. The probe only ever sees the invoking user's processes, so a co-tenant on another account can starve one of our tasks with nothing in the per-task view to show for it. Each node now reports cores= and load1=, printed for any node producing a suspect. It separates the two readings that actually differ: a suspect on a node at load 30 of 32 is contention and exclusion may be justified; the same suspect at 4 of 32 is not. Measured while writing this, the Bayesian campaign's nodes sat at load 8 of 32, holding cores they were not using, which is queue pressure rather than starvation.

Verification

Report-only against that live campaign:

  • All 21 chain tasks report their real footprint (nproc 2-5, ~397%) and classify threaded, against the old nproc=1 med_cpu=0.
  • The two EpiModel-shaped tasks on the same campaign still read nproc=5 med_cpu=99-100 and judge normally, so the standard path is untouched.
  • Forcing CPU_FLOOR=100 makes one a suspect and prints node node1 cores=32 load1=4.17 beside it.
  • Raising MULTICORE_CPU past the chains drops them back into the normal path, so the boundary sits where it is documented.

Also fixes a stderr leak the wider scan exposed: 2>/dev/null placed after an input redirect is set up only after the failing open, so it never caught the shell's own "Permission denied" on unreadable processes.

Version 2.9.0, with NEWS and a new "What this detector can and cannot see" section in the doctor README.

…d tasks

`pgrep -x R` decided which processes belonged to a task, which assumes the work
happens in R. An rstan/cmdstanr task is an idle R wrapper plus a compiled
`model_<hash>` binary running the chains, so the probe reported
`nproc=1 med_cpu=0` for a task consuming four cores. It survived only by
accident, spared by the `nproc < MIN_PROC` startup rule; a second R process, or
a moment between phases, and the doctor would have requeued a healthy Bayesian
fit. Observed on a live 23-task campaign sharing this cluster.

Membership is now decided by SLURM_JOB_ID from /proc/<pid>/environ. Children
inherit it, so the binary is attributed correctly; verified against a running
cmdstan array where the model binary carries both SLURM_JOB_ID and
SLURM_ARRAY_TASK_ID. Login shells and the probe's own processes have no such
variable and drop out for free. The batch script wrapper is excluded by name, so
`nproc` keeps meaning the workload and a starting task keeps its grace period.

Seeing the work is not the same as being able to judge it. The detector rests on
one simulation per core at about 100%, which is what makes a median a starvation
signal and gives the 99-against-51 gap its width. A threaded binary at several
hundred percent has no meaningful median, does not show the ~50% signature, and
carries nothing on the node to say how many threads it asked for, so a healthy
4-thread chain and a contended 8-thread one are indistinguishable from outside.
Tasks whose busiest process exceeds MULTICORE_CPU (default 150, clear of any
single-threaded R process including threaded-BLAS bursts) are reported
`threaded` and skipped. Under-flagging is the right failure here.

Third piece: the probe only ever sees the invoking user's processes, so a
co-tenant on another account can starve one of our tasks with nothing in the
per-task view to show for it. Each node now reports `cores=` and `load1=`, and
that line is printed for any node producing a suspect. It separates the two
readings that actually differ: a suspect on a node at load 30 of 32 is
contention and exclusion may be justified, the same suspect at 4 of 32 is not.
Measured while writing this, the Bayesian campaign's nodes sat at load 8 of 32,
holding cores they were not using, which is queue pressure rather than
starvation.

Verified in report-only mode against that live campaign. All 21 chain tasks now
report their real footprint (nproc 2-5, ~397%) and classify `threaded` instead
of the old `nproc=1 med_cpu=0`; the two EpiModel-shaped tasks on the same
campaign still read nproc=5 med_cpu=99-100 and judge normally; forcing the floor
to 100 makes one a suspect and prints `node node1 cores=32 load1=4.17` beside
it; raising MULTICORE_CPU past the chains drops them back into the normal path,
so the boundary sits where it is documented.

Also fixes a stderr leak the wider scan exposed: `2>/dev/null` placed after an
input redirect is set up only after the failing open, so it never caught the
shell's own "Permission denied" on unreadable processes.
Base automatically changed from dev/doctor-idle-not-starved to main July 29, 2026 11:47
Both conflicts were version numbering only. #62 landed as 2.8.3 rather than the
2.8.2 it was written as, because #61 took that slot first, so NEWS here carried
a stale 2.8.2 header for the same entry. DESCRIPTION keeps 2.9.0, which is the
feature release above it. No code conflicted: #62's classifier changes were
already the base of this branch.

Smoke-tested the merged scripts in report-only mode against the live campaign.
Both feature sets are present and working together: chain tasks report their
real footprint and classify `threaded`, the EpiModel-shaped task still judges
normally, and the confirmation line carries the classification from #62.
@smjenness
smjenness merged commit b2bee67 into main Jul 29, 2026
2 checks passed
@smjenness
smjenness deleted the dev/doctor-multithreaded-workloads branch July 29, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant