Attribute a task's work by SLURM_JOB_ID, and decline to judge threaded tasks - #63
Merged
Merged
Conversation
…d tasks `pgrep -x R` decided which processes belonged to a task, which assumes the work happens in R. An rstan/cmdstanr task is an idle R wrapper plus a compiled `model_<hash>` binary running the chains, so the probe reported `nproc=1 med_cpu=0` for a task consuming four cores. It survived only by accident, spared by the `nproc < MIN_PROC` startup rule; a second R process, or a moment between phases, and the doctor would have requeued a healthy Bayesian fit. Observed on a live 23-task campaign sharing this cluster. Membership is now decided by SLURM_JOB_ID from /proc/<pid>/environ. Children inherit it, so the binary is attributed correctly; verified against a running cmdstan array where the model binary carries both SLURM_JOB_ID and SLURM_ARRAY_TASK_ID. Login shells and the probe's own processes have no such variable and drop out for free. The batch script wrapper is excluded by name, so `nproc` keeps meaning the workload and a starting task keeps its grace period. Seeing the work is not the same as being able to judge it. The detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal and gives the 99-against-51 gap its width. A threaded binary at several hundred percent has no meaningful median, does not show the ~50% signature, and carries nothing on the node to say how many threads it asked for, so a healthy 4-thread chain and a contended 8-thread one are indistinguishable from outside. Tasks whose busiest process exceeds MULTICORE_CPU (default 150, clear of any single-threaded R process including threaded-BLAS bursts) are reported `threaded` and skipped. Under-flagging is the right failure here. Third piece: the probe only ever sees the invoking user's processes, so a co-tenant on another account can starve one of our tasks with nothing in the per-task view to show for it. Each node now reports `cores=` and `load1=`, and that line is printed for any node producing a suspect. It separates the two readings that actually differ: a suspect on a node at load 30 of 32 is contention and exclusion may be justified, the same suspect at 4 of 32 is not. Measured while writing this, the Bayesian campaign's nodes sat at load 8 of 32, holding cores they were not using, which is queue pressure rather than starvation. Verified in report-only mode against that live campaign. All 21 chain tasks now report their real footprint (nproc 2-5, ~397%) and classify `threaded` instead of the old `nproc=1 med_cpu=0`; the two EpiModel-shaped tasks on the same campaign still read nproc=5 med_cpu=99-100 and judge normally; forcing the floor to 100 makes one a suspect and prints `node node1 cores=32 load1=4.17` beside it; raising MULTICORE_CPU past the chains drops them back into the normal path, so the boundary sits where it is documented. Also fixes a stderr leak the wider scan exposed: `2>/dev/null` placed after an input redirect is set up only after the failing open, so it never caught the shell's own "Permission denied" on unreadable processes.
Both conflicts were version numbering only. #62 landed as 2.8.3 rather than the 2.8.2 it was written as, because #61 took that slot first, so NEWS here carried a stale 2.8.2 header for the same entry. DESCRIPTION keeps 2.9.0, which is the feature release above it. No code conflicted: #62's classifier changes were already the base of this branch. Smoke-tested the merged scripts in report-only mode against the live campaign. Both feature sets are present and working together: chain tasks report their real footprint and classify `threaded`, the EpiModel-shaped task still judges normally, and the confirmation line carries the classification from #62.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #62. Review that one first; the base here is
dev/doctor-idle-not-starved, and I will retarget tomainonce it merges.pgrep -x Rdecided which processes belonged to a task, which assumes the work happens in R. Anrstan/cmdstanrtask is an idle R wrapper plus a compiledmodel_<hash>binary running the chains, so the probe reportednproc=1 med_cpu=0for a task consuming four cores. It survived only by accident, spared by thenproc < MIN_PROCstartup rule. A second R process, or a moment between phases, and the doctor would have requeued a healthy Bayesian fit. Observed on a live 23-task campaign sharing this cluster.Three changes
Membership by
SLURM_JOB_ID, not by process name. Children inherit it, so the compiled binary is attributed correctly. Verified against a running cmdstan array where the model binary carries bothSLURM_JOB_IDandSLURM_ARRAY_TASK_ID. Login shells and the probe's own processes have no such variable and drop out for free. The batch script wrapper is excluded by name, sonprockeeps meaning the workload and a starting task keeps itsMIN_PROCgrace period.Decline to judge threaded tasks. Seeing the work is not the same as being able to judge it. The detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal and gives the 99-against-51 gap its width. A threaded binary at several hundred percent has no meaningful median, does not show the ~50% signature, and carries nothing on the node to say how many threads it asked for, so a healthy 4-thread chain and a contended 8-thread one are indistinguishable from outside. Tasks whose busiest process exceeds
MULTICORE_CPU(default 150, clear of any single-threaded R process including threaded-BLAS bursts) are reportedthreadedand skipped. Under-flagging is the right failure here.Report the node. The probe only ever sees the invoking user's processes, so a co-tenant on another account can starve one of our tasks with nothing in the per-task view to show for it. Each node now reports
cores=andload1=, printed for any node producing a suspect. It separates the two readings that actually differ: a suspect on a node at load 30 of 32 is contention and exclusion may be justified; the same suspect at 4 of 32 is not. Measured while writing this, the Bayesian campaign's nodes sat at load 8 of 32, holding cores they were not using, which is queue pressure rather than starvation.Verification
Report-only against that live campaign:
nproc2-5, ~397%) and classifythreaded, against the oldnproc=1 med_cpu=0.nproc=5 med_cpu=99-100and judge normally, so the standard path is untouched.CPU_FLOOR=100makes one a suspect and printsnode node1 cores=32 load1=4.17beside it.MULTICORE_CPUpast the chains drops them back into the normal path, so the boundary sits where it is documented.Also fixes a stderr leak the wider scan exposed:
2>/dev/nullplaced after an input redirect is set up only after the failing open, so it never caught the shell's own "Permission denied" on unreadable processes.Version 2.9.0, with NEWS and a new "What this detector can and cannot see" section in the doctor README.