Proxy Updates - #1
Open
A-Tarraf wants to merge 46 commits into
Open
Conversation
…isplay input flexibility
… a bug where invalid args were sent to ftio
…o, moved ftio visualization to mean value and changed ftio signal name to proper wave name
… ability in case of node failure
…mq communication and fixed 2 small bugs
Feature ftio
Author
|
We tested out the version and it work great |
… expand - New --auto-root / --root-url-dir flags: child proxies discover the root URL from a shared filesystem file (root.url) written by the root at startup. Root URL can also be injected via PROXY_ROOT_URL env var. - Graceful leave: SIGTERM handler sends /leave?from=<url> to the root before exiting, triggering immediate TBON repair without waiting for a missed scrape. - /leave HTTP endpoint on the root: removes the departing node from the topology and calls the existing self-repair logic to rewire the TBON. - Fix NaN/null serialization: Gauge min/max/total fields that are NaN were serialized as JSON null, crashing the root when deserializing child scrapes. Fixed with a serialize_with helper that maps NaN/infinite to 0.0; the binary UNIX socket protocol (which does not support deserialize_any) is unaffected. - Add experiment/run_malleability_test.sh: end-to-end test against a 4-node Docker cluster exercising graceful leave, shrink self-repair, and expand auto-join. Script checks prerequisites and auto-builds the binary if needed. - Add experiment/README.md with quick-start, cluster setup link, expected output, and a table of what each step tests. - Update README.md with a Malleability Support section documenting all new flags, endpoints, DMR integration guidance, and a pointer to the experiment.
Root cause: {{fnall}} in mpi_wrappers.w generates Fortran wrappers for
ALL MPI functions. OpenMPI 5.x added the MPI-4 Session API
(MPI_Session_*), whose Fortran wrappers use MPI_Session_f2c() — but that
returns an int handle while the C functions expect an MPI_Session*
pointer. GCC 15 treats this as an error.
Fixes:
1. exporters/mpi/mpi_wrappers.w — Added 9 problematic MPI-4 Session
functions to the {{fnall}} exclusion list (they handle process-set
management, not data transfer, so excluding them doesn't affect proxy
measurement)
2. install.sh — Changed the mpicc step to use || error_out instead of
checking file existence, so a real compilation failure is caught
and reported
Web UI (ftio.html / trace.html): - Add Parallel Analysis tab: run FTIO on all metrics of a stored job at once; show dominant frequency scatter, heatmap, correlation matrix, category waves, and per-metric selection plot - Correlation matrix uses full width, initial height capped at 2400 px - Per-plot height sliders, click-to-activate scroll zoom - Selection plot: checkboxes for dominant freq / reconstructed / raw data - Category waves: raw metric overlay, time axis covers full t_end range - Metric table: search field, fixed-height scrollable container - Job selector defaults to most-recent job; shows trace size - Real-time processed/total counter via /ftio/progress polling Rust fixes: - trace.rs: replace .unwrap() with ? in offset_of_last_frame_start to avoid panic on empty or truncated trace files - trace.rs: revert chunked ZMQ back to single-batch send; chunking added N sequential round-trips with no parallelism benefit (Python ProcessPoolExecutor already parallelises within one call) - proxywireprotocol.rs: sanitize NaN/Inf to 0.0 on serialization so remote proxies never receive JSON null where f64 is expected - exporter.rs / ftio.rs: demote spurious log::error!/warn! to log::info! for normal startup and cleanup paths Tests: - proxywireprotocol::tests: NaN/Inf counter and gauge serialize to 0.0, finite values pass through unchanged, counter merge accumulates - trace::tests: empty and truncated files return Err (not panic), TraceState::new creates a valid scannable file, single-frame offset is 0 Docs: - README.md: rewrite with flags table, HPC/SLURM deployment section, web UI overview, endpoint reference; remove Docusaurus-only markup - docs/start_root_proxy.sh: script to start root proxy on login node - docs/sbatch_example.sh: annotated SLURM job template with 3-step guide
…ovements - Synthesize ___bandwidth___<fn> Gauge (bytes/s = Δsize/Δtime_in_call) for every size/time counter pair at each scrape; forwarded to FTIO automatically - Add --no-bandwidth flag to disable the synthesis - Fix procs badge URL (Proxy scrapes stored target_url with /job suffix) - Fix auto-latest dropdown closing while user scrolls metrics - Fix install.sh: prefer release over debug, respect MPICC/PYTHON env vars, pass -c mpicc to wrap.py, fix build dir detection - Add --strace-only install flag for faster strace-only rebuilds - Docker monitoring doc rewritten: shared-volume install (/opt/hpc/build/metric-proxy), FTIO setup with host-wrapper approach, HACC-IO test walkthrough
- FTIO analysis now runs in a background thread (was inline in the sequential scraping loop, starving all trace scrapers: jobs got a single data point and appeared late on the root) - FTIO restricted to real user jobs (was every main/Node trace on all proxies every 10s, saturating the host) with one analysis in flight per proxy (guard on FtioClient) - 5s timeout on proxy-to-proxy HTTP scrapes (hung peer froze the loop) - relax_tracked_jobs(): scraper teardown releases its jobs on the root - trace.html: auto-refresh no longer disabled while a select has focus; metric list re-fetched every cycle (DOM rebuilt only on change); jobs sorted newest-first, auto-latest on by default - new proxy_mpi_ranks metric (one per MPI rank, via mpi___ Desc detection) + /ranks endpoint; UI badge shows ranks and procs - docs: FTIO-in-container recipe, ranks vs procs, speed & validation report (fork vs upstream benchmark, HACC-IO overhead breakdown: proxy free, strace exporter 5-9x on syscall-dense jobs) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4UvJCZzuEQdGvbJ9KdrDJ
…tracking - Replace the stored ___bandwidth___ gauge (per-rank in-call speed, spike at burst completion) with the virtual ___bandwidth_dewrap___<fn> metric: bursts are spread back over their wall-clock span (dt / concurrent ranks), values anchored at interval start with sample-and-hold; computed on the fly at plot time, trace stays append-only. Remove --no-bandwidth. - Default FTIO custom args to --dewrap so dewrapped analyses run out of the box (FTIO adds the reconstructed series on top, keyed to match the UI virtual metric). - Add job_mpi_ranks: live per-job rank count from rank connections (counted on mpi___ Desc, decremented on disconnect, retroactive job attribution), pushed with set semantics per node, summed at root - integer counts that follow malleable jobs (verified 4->8->4). - Relabel UI ranks/procs badge as proxy-wide; document metric semantics (interception at call return, rank-summed time, dewrap, counter vs gauge aggregation, FTIO communication) in docs/metrics_semantics.md.
…per-job rank badge Bug fixes, each verified on the 9-node docker cluster: - Aggregation double-count: when a proxy scraped a child's /job, the first-seen snapshot of every counter was counted twice (push() initialized the new entry WITH the value, accumulate() then added it again), so each aggregation hop reported child_now + child_first_seen. Invisible for byte counters (~0 at job registration) but job_mpi_ranks showed 15 for a 4-rank job through a 3-level scrape tree. Counters are now declared with a zeroed value and imported exactly once; verified: root reports exactly 4 ranks with a clean 1-2-3-4 startup ramp. - Counter delta/merge timestamps: delta() subtracted timestamps and merge() averaged them, producing garbage ts on aggregated counters; delta keeps the new sample's ts, merge keeps the freshest. - Hold direction in the trace UI: stepped mode used Chart.js 'after', which holds the NEW value backward across the interval (bursts appeared one bin early); now 'before' - each value holds from its own timestamp until the next sample. - derivate_time_serie now anchors each rate at the interval START with a trailing 0 (sample-and-hold, consistent with the dewrapped bandwidth convention) instead of the interval end. UI: the ranks/procs badge next to the Target selector now shows the selected job's live MPI rank count (job_mpi_ranks via /job, with the last traced value for ended jobs); proxy-wide totals only for Node:/main. Docs: FTIO quick-start step clarified (the proxy normally spawns FTIO itself; the manual host-side start is Docker-specific), Case 2 explains what runs where and what a proxy restart is, start.sh preserved verbatim in an appendix (local-only modifications), and a measured warning against pinning proxies with taskset on the docker cluster (1:49 unpinned vs 6-19 min pinned; on real Slurm clusters the allocation reserves the proxy's core instead).
- profiles.rs: refresh_profiles now skips a .profile that fails to parse (truncated JSON left by a proxy killed mid-write) with a warning and continues, instead of aborting the whole startup load and crashing. - exporter.rs/webserver.rs: make /join and /pivot idempotent against the current tree so a node cannot acquire two parents. A double-parented node had its counters summed twice at the root (a 4-rank job showed 8-16 ranks and ~2x bytes) - the real cause of the stuck job_mpi_ranks and doubled HACC byte totals. - docs/docker_monitoring.md: verify topology via /topo, pinning guidance.
Ignore Rust target/, runtime traces/, Python __pycache__, and autoconf regeneration byproducts (*~, .version) so they stop cluttering status.
The strace exporter was launched as `proxy_exporter_strace -c -- <app>`, with no
syscall filter, so it ptrace-stopped the application on *every* syscall. Each
traced syscall costs ~50 us (two stops, entry + exit), so the price is set by the
application's syscall count -- not by the sampling period. Lowering -S never
helped, and could not have.
strace is also active by default (proxy_run runs all detected exporters unless
restricted with -e), so a plain `proxy_run -- app` was paying this in full.
Add `-T/--strace-trace` to proxy_run, selecting which syscalls are traced. When a
filter is active the exporter runs with --seccomp-bpf (and -f, which it requires),
so non-matching syscalls are dropped by the kernel and never stop the tracee. The
default is a denylist of nine hot, uninformative syscalls that MPI progress
engines spin on:
!getpid,gettid,futex,sched_yield,clock_gettime,clock_nanosleep,
nanosleep,rt_sigprocmask,rt_sigaction
This is a denylist, not an I/O allowlist: file, network, memory, process and
signal syscalls are still traced. Measured (100k write + 500k getpid):
proxy_run -T all -- app 23.0 s 43 metrics (previous behaviour)
proxy_run -- app 4.6 s 41 metrics (new default)
proxy_run -e mpi -- app 0.29 s -- (no ptrace; ~untraced)
Only the denylisted syscalls lose their hits/time counters; all byte counters are
identical. `-T all` restores the exact previous command line.
Also buffer the exporter->proxy socket: MetricProxyClient::send() serialized JSON
straight onto a raw UnixStream, one message per counter, costing ~41 sendto(2)
calls per counter. A BufWriter flushed once per dump takes a 72-counter push from
29,622 sendto to 13, and makes it flat in counter count (it previously saturated
the push thread at N=1000). This is an efficiency fix, not an application-overhead
one -- it does not measurably change application wall time.
Docs:
- docs/strace_exporter.md cost model, the filter, exactly what the default drops
- docs/cluster_benchmark.md how to measure exporter overhead on a cluster
- docs/sbatch_overhead_sweep.sh ready-made sweep job (placeholders for site config)
- docs/paper_plan.md contributions and experiment plan
The sweep guards against a nasty failure mode: a leaf proxy given -r whose root is
unreachable calls exit(1), leaving no proxy on the node. metric_proxy_init() then
fails, every counter handle is NULL, and the application runs UNINSTRUMENTED at
full speed -- a fast, wrong, convincing number. The script pre-flights the root and
re-checks after every run that the job was actually seen.
The prefix is required, but the usage string wrote it as [PREFIX], which by convention means optional. Spell it <PREFIX>, and show what the script installs where, plus the PATH/LD_LIBRARY_PATH exports needed afterwards.
wrap.py did `from numpy import size`, but the imported name is never used -- the only size() call is sizer.size(), a method on a different object. The dead import made numpy a hard build dependency, so install.sh failed with ModuleNotFoundError on any system whose python3 lacks it (e.g. a bare cluster module). Verified the generated wrappers are byte-identical with and without the import.
A second proxy on a node (leftover from a cancelled job, or a leaf landing on the root's node) unlinked the live proxy's socket, bound its own, then died on AddrInUse -- leaving a stale socket. The survivor kept serving HTTP and looked healthy, while every exporter on that node got ECONNREFUSED and its application ran UNINSTRUMENTED at full speed. That produces a fast, wrong number. Bind the HTTP port first and refuse to start if taken, before touching the socket. --no-ftio: skip the FTIO server probe and per-job analysis. Without it a proxy with no FTIO server stalls ~5 s at startup and logs "FTIO client address not set" on every trace cycle forever. Sweep script: its guard polled /joblist, which does not exist (route is /job/list), so every run was flagged NO-UNINSTRUMENTED regardless of reality. /job/list is wrong anyway -- it lists only LIVE jobs, always empty once a run ends. Replaced with two checks: the app's output has no "Not Connected to Metric Proxy" (reached its local proxy), and the root's /metrics counter sum increased (data reached the root). Also kills stray proxies first and polls every node.
Step-by-step, as actually run on Lichtenberg 2026-07-14: build, export the tool path (the only env setup needed), verify the -T / --no-ftio flags, and verify seccomp-BPF really engages (measured: 2.043s / 201,990 syscalls unfiltered vs 0.119s / 1,881 filtered -- 17x, with getpid never trapping). Also: how to read local_ok/remote_ok in the CSV, why /job/list must not be used as an instrumentation check (it lists only LIVE jobs), and the two-proxies-on-one- node failure that silently leaves a node uninstrumented.
The tree-repair path unwrapped an HTTP query to the replacement node's /period.
That runs exactly when nodes are disappearing, so the replacement is often gone
too. The panic fired while holding known_client, poisoning the mutex -- every
later lock().unwrap() then panicked with PoisonError and /remove returned 500
forever, so the root could never repair its tree again.
Fall back to the default period instead of unwrapping, and make the five
known_client lock sites poison-tolerant so one panic cannot wedge the tree.
Verified against a dead node: /remove now returns 200 (was 500), no panics, and
the root keeps serving /job/list, /join/list and /metrics.
Sweep script: export LC_ALL=C. In a comma-decimal locale printf rejected the
timings ("25.331047370: invalid number") and the per-run lines printed as
integers. The CSV and summary were unaffected.
…trace-tab UX
Build
- exporters/mpi/mpi_wrappers.w: {{fnall}} generated Fortran PMPI wrappers
for every MPI symbol. OpenMPI 5.x added the MPI-4 Session API, whose
generated wrappers pass MPI_Session_f2c()'s int through where an
MPI_Session* is expected; GCC 15 makes that a hard error. Exclude the 9
MPI_Session_* / MPI_Group_from_session_pset functions (process-set
management, not data transfer, so proxy measurement is unaffected).
- install.sh: mpicc step now fails loudly (|| error_out) instead of just
checking for the output file.
- install.sh: build the vendored strace with --enable-bundled=yes
--disable-gcc-Werror so it compiles against modern kernel headers
(io_uring_buf_reg static_assert) without -Werror aborting on unrelated
header drift.
- install.sh: --strace-only was broken (SOURCE_ROOT only set in the
full-build path) and always re-ran the non-idempotent ./bootstrap over
an already-generated tree. SOURCE_ROOT is computed unconditionally and
bootstrap runs only when ./configure is absent.
FTIO backend
- The periodic scrape already serialised FTIO calls with ftio_client
.in_flight, but /ftio/run_all and /ftio/run_metric did not — a manual
"Run Analysis" could race the background scrape's ZMQ REQ/REP and hit a
send timeout ("Resource temporarily unavailable"). Both handlers now
take the guard and return a clear "already running" response.
- generate_fallback_ftio_model parsed admire_proxy_invoke_ftio's stdout by
the first '[', which can land inside diagnostic text the tool prints
before its result (e.g. "With Arguments: ['--freq', ...]"). Take the
last line that actually starts with '[' (its single json.dumps output).
Web UI (trace.html)
- "Modify FTIO parameters" is now a <details> panel (like ftio.html's
Parallel Analysis) instead of a checkbox that toggled a div.
- Overrides are no longer gated on the panel being expanded: editing any
field applies it live (debounced) and it stays applied after collapse;
"Restore" reverts to the proxy defaults. Previously the old "Apply"
tick only flipped a flag and the request went out on the next poll.
- update_ftio() falls back to an on-demand /ftio/modified_args compute
when /trace/ftio has no cached model yet, so the trace tab no longer
shows nothing while waiting for a background scrape.
- "No dominant frequency" messages recoloured from red to the same warn
style as the low-confidence case, with a hint to lower
--sampling-period/-S (Nyquist).
README
- Corrected the FTIO HTTP endpoints (POST /ftio/run_all, /ftio/run_metric,
GET /ftio/all_models, /ftio/logs); the documented GET /ftio/run no
longer exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LTMkWDXn6fkw52in397QLv
Brings the malleability work (0542b93) onto development: --auto-root / --root-url-dir root.url discovery, PROXY_ROOT_URL, the graceful-leave SIGTERM handler + /leave endpoint, and the repair_tbon() extraction. Conflict resolution: - src/proxywireprotocol.rs: both branches added the same NaN/inf-as-0.0 serialize helper under different names. Kept development's (ser_f64_sanitized), which the rest of development already references. - src/webserver.rs: main refactored the inline TBON-repair logic out of handle_remove into repair_tbon() and reused it from handle_leave. development had bug-fixed that same logic inline (0d4acc1: period fallback instead of a panic that poisoned known_client). Took main's repair_tbon() structure — it already has the period fallback — and applied development's lock-poison recovery (lock().unwrap_or_else(|e| e.into_inner())) to repair_tbon() and handle_leave for consistency with the rest of the file. - install.sh: kept development's `fi # end if STRACE_ONLY = 0`. - README.md: kept development's trailing separator and single Acknowledgments section; inserted main's Malleability Support section. - Cargo.lock: took main's (adds ctrlc), reconciled by cargo build. Verified: cargo build --release + cargo test pass; merged binary exposes both --no-ftio and --auto-root, serves /leave, and writes root.url. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LTMkWDXn6fkw52in397QLv
…reached root)
ExporterFactory::set_data normalizes factory.root_proxy to 'http://host:port',
but the SIGTERM graceful-leave handler did format!("http://{}/leave?...", root_url),
producing 'http://http://host:port/leave?...'. reqwest could not resolve that, the
error was swallowed (let _ = ...), and the node was only removed later by the
scrape-timeout fallback — defeating the point of graceful leave.
Local 4-process test (no Docker): with a 3s scrape period the child is now
removed 0.15s after SIGTERM (was ~3s); shrink self-repair and --auto-root
expand also verified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LTMkWDXn6fkw52in397QLv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The new version of FTIO creates a prediction server that can be accessed via ZMQ and MessagePack.
Tim Deringer (@Tim-Dieringer) just finished his thesis on some improvements and linkage with FTIO. More precisely, these are his changes:
The changes to FTIO's integration are complementary to https://github.com/tuda-parallel/FTIO/tree/feature/metric_proxy_bindings