feat(cpu): add cpu_power sampler for RAPL energy and C-state residency

[yangxi]

Summary

• Adds a cpu_power sampler reporting CPU energy, power, and idle-state residency, read entirely through perf (no /dev/cpu/N/msr).
• PMUs (power, power_core, cstate_core, cstate_pkg) are discovered from /sys/bus/event_source/devices with no CPU vendor or model detection - each PMU's cpumask supplies both the permitted CPUs and the counter's scope, so only the domains and C-state levels a part actually implements are reported.
• Adds Power and C-State Residency sections to the CPU dashboard, plus power/energy unit systems. Every subgroup is gated on the metric being present, so the section adapts per host.

Details

Package-scope metrics are indexed by the CPU's ordinal position in the PMU cpumask rather than its CPU id: the ids are sparse (0,64 on a two-socket host) while the metric groups are sized to a new MAX_PACKAGES = 8.

Two choices worth calling out:

• perf, not MSR. The power PMU covers every implemented domain, needs CAP_PERFMON rather than CAP_SYS_RAWIO, handles the 32-bit counter wrap, and reports pre-scaled energy. The msr PMU is not an alternative route to RAPL - despite a 64-bit config, msr_event_init() rejects anything outside a fixed allowlist of eight architectural counters, none of them energy registers.
• Dashboard plots derive watts from the energy counters, not the cpu_*_power gauges. The gauges average over the sampler's own refresh interval, so a scrape landing mid-interval returns a stale value, and being integer milliwatts they read 0 for a microwatt-scale domain (an idle iGPU). irate(cpu_<domain>_energy) / 1e6 is correct over any window. The gauges are still exported and documented with that caveat.

Test plan

• cargo build, cargo clippy --all-targets, cargo fmt --check clean (one pre-existing unrelated too many arguments warning in crates/dashboard)

• cargo test - 587 + 7 + 5 pass, 0 failures

• node --test tests/*.mjs - all pass except wasm_viewer_histogram_kpis, which needs a prebuilt site/viewer/pkg/ artifact absent locally (fails identically on a clean tree)

• scripts/check-viewer-symlinks.sh - 57 shared modules resolve

• Validated against turbostat on three architecturally distinct hosts under a staged sysbench load (idle, 1 thread, half cores, all cores, idle), comparing mean package power over the same window:

  host	turbostat PkgWatt	rezolus	delta
  sky (Skylake, 12c)	36.31 W	36.90 W	1.6%
  amd (Zen, 24t/12c)	80.22 W	81.15 W	1.2%
  hyb (hybrid, 16t)	27.79 W	28.05 W	0.9%

• Overhead measured (n=600 per host, scrape-driven): per-refresh p50 of 146 us for amd's 13 counters, 451 us for sky's 28, 1002 us for hyb's 37 - roughly 11-27 us per counter, each being one read() on a perf fd. cpu_frequency measured 316 us / 585 us on amd / hyb in the same windows for comparison. A/B against the same scrape load with the sampler disabled puts its marginal process cost at +0.28 to +1.84 pp of one core at 10 scrapes/s.

• Discovery verified per host: sky 28 counters, amd 13, hyb 37 - with zero vendor detection. The hybrid host is the interesting case: core_c7_residency covers only its 6 P-cores while core_cstate_residency covers all 10 physical cores, which is why the summed metric exists.

• Multi-socket path (ordinal indexing, per-package heatmaps) is untested - all three hosts are single-package.

Generated with Claude Code

[brayniac]

Reviewed against the PMU work that landed today (#1091, #1093, #1096). One substantive finding, one correction to something I said earlier, and a rebase note.

Finding: max(index) + 1 is a dense prefix, and these readers can be sparse

let bound = |f: fn(&Kind) -> bool| {
    readers.iter().filter(|r| f(&r.kind)).map(|r| r.index + 1).max().unwrap_or(0)
};
CPU_POWER_ENERGY_ACQ.set_member_bound(bound(...));

The comment above it cites principle 18 and is right to — but set_member_bound(n) means the indices 0..n, and discover() only pushes a Reader when the counter opens:

let Some(counter) = open_counter(pmu, event, cpu) else { continue; };

So the populated indices need not be contiguous. Two ways in, both real:

• power_core/energy-core uses Index::Cpu — the CPU id. A host with offline or non-contiguous CPU ids gives sparse indices by construction.
• Any single open_counter failure (permissions, a core that will not open) leaves a hole while a later index still succeeds, so max + 1 spans it.

Why that matters more than it looks: an unwritten CounterGroup slot reads as 0, not as absent. The gaps would publish zero joules, and irate() over them zero watts — a wrong value rather than missing data, on a metric whose whole point is measuring draw.

We hit exactly this today in #1096: a CPU-masked PMU reservation opened counters on 16 of 32 CPUs, declared the prefix, and published cpu_cycles = [0, 0, 0, 0] on the CPUs it had skipped.

The fix is now one line. #1096 added AcquisitionGroup::set_member_set(&[usize]), which declares the exact indices instead of a prefix:

let indices = |f: fn(&Kind) -> bool| -> Vec<usize> {
    readers.iter().filter(|r| f(&r.kind)).map(|r| r.index).collect()
};
CPU_POWER_ENERGY_ACQ.set_member_set(&indices(|k| matches!(k, Kind::Energy { .. })));

It sorts and de-duplicates, and is clamped to the backing array like the bound is. The prefix form is still correct where the population genuinely is 0..n; this is for where it is not.

Correction: this does not need a PMU budget entry

I said earlier that a perf-based sampler would need a PmuKind entry in pmu::DEFAULT_PRIORITY or it would claim counters outside the budget. That was overstated for this sampler. RAPL lives on its own PMU — /sys/bus/event_source/devices/power is type 18 against the core PMU's type 4 — so it does not draw on the contended core general-purpose counters at all. Being unbudgeted is the correct outcome, exactly as for cpu_l3 (uncore) and cpu_frequency (MSR).

Optional: adding it to the table as PmuKind::Rapl (or reusing Msr) would keep that table the complete picture of who opens hardware events and where, which is what its doc claims — but it changes no behaviour, so take it or leave it.

Rebase note

This branch predates #1085, #1093 and #1096. Worth rebasing before merge — in particular #1085 now bounds any acquisition group whose sampler did not come up to zero members, so cpu_power's groups must be registered under a name that appears in SAMPLERS. The Linux-only every_registered_group_names_a_real_sampler test will catch it if not.

[brayniac]

Reviewed against the PMU work that landed today (#1091, #1093, #1096).

One substantive finding, inline below: the member bound is a dense prefix, and these readers can be sparse — so the gaps would publish zero joules rather than nothing. Same failure #1096 fixed today, and there is now a one-line way to express the right thing.

Two notes that need no change:

This does not need a PMU budget entry, and I was wrong to imply otherwise. RAPL is its own PMU — /sys/bus/event_source/devices/power is type 18 against the core PMU's type 4 — so it never draws on the contended core general-purpose counters that #1093 rations. Being unbudgeted is correct here, exactly as for cpu_l3 (uncore) and cpu_frequency (MSR). Adding it to pmu::DEFAULT_PRIORITY as a non-Core kind would only keep that table the complete picture of who opens hardware events, which is what its doc claims — behaviour-neutral, so take it or leave it.

Rebase before merge. This branch predates #1085, #1093 and #1096. #1085 in particular now bounds any acquisition group whose sampler did not come up to zero members, so cpu_power's groups must be registered under a name that appears in SAMPLERS; the Linux-only every_registered_group_names_a_real_sampler test will catch it if not.

The sampler itself reads well — the Index::Ordinal / Index::Cpu distinction and the cpumask handling are the parts I expected to find wrong and did not.

[brayniac]

Nice sampler — the discovery path, the sparse-index registration, and especially the measured overhead (n=600 across three hosts, with cpu_frequency as a comparison baseline) are exactly what docs/principles.md asks for. Principle 16 is satisfied without argument, which is the part that usually isn't.

Requesting changes on two things that are wrong rather than merely wide: the acquisition-group handling, and three dashboard percentages that are silently scaled wrong on any SMT host.

---

Acquisition groups / observation windows

The group abstraction is declared here but not honored on either axis — time or membership.

W1 — Five groups, one identical window. refresh() acquires all five guards up front (src/agent/samplers/cpu/linux/power/mod.rs:196), sweeps a single flat Vec<Reader>, then finishes all five (:244). All five windows are byte-identical spans of the whole sweep. Principle 18 names this exact outcome — "several groups all wearing one identical whole-loop window: no new information, pure schema bloat" — as the reason the device-visit archetype collapses to one group. As written this pays five groups' schema cost for one group's information.

It also contradicts its own doc comment: Reader.acq (:78-81) is documented as held "so refresh brackets each entity space separately", but refresh never reads the field. init already partitions readers by group at :146, so group-major bracketing is a few lines.

W2 — The stale-value dispensation is claimed without meeting its condition. The "None discard" comment (:188-190) cites the cpu_l3/cpu_dtlb ruling. That ruling is real, but cpu_l3 and cpu_dtlb each declare exactly one group (CPU_L3_ACQ, CPU_DTLB_ACQ), so their single bracket genuinely is their single read section. Principle 18 scopes the "stale value under a freshly stamped window" trade to the device-visit archetype and only when "(c) the families genuinely cannot be read family-major without re-visiting every device once per family." Here (c) plainly fails — a flat Vec of independent perf fds, no per-device handle, no re-visit cost. So this takes the archetype's cost without having its constraint.

W3 — The first refresh stamps a fresh window over values nothing wrote. Three continue paths skip the write (failed read(), the baseline sample, counter went backwards) while finish() stamps regardless. On the very first refresh every member is in the baseline path, so every counter reads 0 under a freshly stamped window. By this PR's own reasoning at :132-134 — "a declared index nothing writes would publish zero joules -- a wrong value on a metric whose whole purpose is measuring draw" — that's the same defect the sparse-index registration carefully avoids, just transient rather than permanent.

W4 — Membership is a union across each group's metrics, so absent domains come back as phantom series. init sets one member set per group (:143-151), and the snapshot builder applies that same set to every metric routed to the group (src/agent/exposition/http/snapshot.rs:1188-1196, member_set = ag.member_set()). But CPU_PACKAGE_ENERGY, CPU_CORES_ENERGY, CPU_IGPU_ENERGY and CPU_DRAM_ENERGY all share cpu_power_package_energy, and CORE_C1..C10 all share cpu_power_core_cstate. So a server exposing only energy-pkg/energy-ram still emits cpu_igpu_energy and cpu_cores_energy at every package ordinal, and a Skylake exposing only c1/c3/c6/c7 still emits core_c2/c8/c9/c10_residency for every core. That contradicts docs/metrics.md's "the rest are absent rather than zero" and the stats.rs comment about fabricated zeros, and it makes the dashboard render empty "Integrated Graphics Power" / "Core C2" subgroups.

W1 and W4 have a single fix: one group per RAPL domain and per C-state level (principle 18's "families do not collapse"), each bracketed group-major over just its own readers. That yields genuinely distinct windows and correct membership. Per-metric membership within one group isn't something the machinery supports today, so splitting the groups is the path.

---

Dashboard: three percentages are scaled wrong on any SMT host

crates/dashboard/src/dashboard/cpu.rs:434, :466, :501 — one root cause.

sum(irate(core_cstate_residency[5m])) / sum(irate(cpu_tsc[5m])) sums the numerator over the core-scope cpumask — docs/metrics.md:177 states that PMU is one CPU per physical core with SMT siblings excluded — against a denominator written per logical CPU (CPU_TSC.set(self.id, tsc), src/agent/samplers/cpu/linux/frequency/mod.rs:228).

• A fully idle 2-way SMT box reads 0.50, not 1.00.
• The hybrid host in your own test matrix (10 physical cores / 16 threads) reads 0.625.
• :501 is worse: package_cN_residency is a single per-package counter ticking at TSC rate, divided by the TSC sum over all logical CPUs — a package sitting 100% in PC6 renders as 8.3% on a 12-thread single-socket host. :512 has the same scaling error.

Every one of these is .percentage_range(), so the result looks plausible rather than obviously broken. The sum by (id) / sum by (id) companions at :444 and :476 are correct — the join drops unmatched ids — which is exactly what hides the aggregate's mismatch in testing.

---

Smaller items

• energy-psys double-counts on multi-socket (power/mod.rs:297-303). It uses Index::Zero while its sibling power-PMU domains use Index::Ordinal precisely because that cpumask has one CPU per package. On a two-socket part exposing PSYS, both readers add() into cpu_platform_energy{id="0"} → 2×. PSYS is platform-wide, so that's the same quantity twice, not two halves. You note multi-socket is untested; either open PSYS on the first cpumask entry only, or index it by ordinal.
• Missing DOMAIN_ALIASES entries (src/analysis/extract/context.rs:327-330). domain_of("core_c6_residency") → "core" and domain_of("package_c6_residency") → "package"; neither aliases to "cpu". On a non-"rezolus" source those land in uncertain_domains, so cpu_power escapes build_coverage's pruning and gets reported in subsystems_absent while its own metrics are present in the recording. Needs ("core","cpu") and ("package","cpu") — the same pattern as drive→drivehealth.
• Index::resolve warns on an over-capacity index but still builds the reader (power/mod.rs:352). Above MAX_PACKAGES = 8 that's a read() syscall per event per refresh whose result is always discarded. Returning None would make the ceiling visible as "no data" instead of silent per-tick work.
• Lone half-width plots. When per_id is false (single-package — the common case) the domain's only plot uses plot_promql, leaving a half-empty row; cpu.rs uses plot_promql_full for exactly this at :38, :317, :347. Same at :397, :433, :463, :498.
• Stale references to removed gauges. The comment at cpu.rs:361 and the PR description both refer to cpu_*_power gauges as "still exported", but no power gauge exists in stats.rs (and docs/metrics.md correctly says there is no power metric). The module header still opens "Collects CPU energy, power, and idle-state residency". Unit::Energy (plot.rs:580) and the energy unit system in units.js are added but unused by any plot.

---

What's already right

Stamp-last ordering, one writer (the Mutex-serialized refresh), sparse index registration declared explicitly rather than as a 0..n prefix (with a good comment on why), like-entity grouping within each declared group, graceful absence under virtualization, and the measured overhead. The sum by (id) variants of every affected plot are correct as written.

🤖 Review assisted by Claude Code

[yangxi]

Thanks — all of it was right, and the two blocking findings were wrong rather than wide as you said. Pushed as fa04be6.

Acquisition groups (W1–W4)

Went further than one group per RAPL domain and per C-state level: one group per metric, 23 in total. The reason is W4's mechanism — snapshot.rs applies ag.member_set() per metric, not per group, so any group backing more than one metric declares the union of their populations and the short ones fabricate zeros at the surplus indices. Splitting by domain/level happens to give 1:1 here anyway, so this is the same split stated in the terms that actually make it correct.

• W1 — init partitions the readers once (extending the :146 split you noted); refresh iterates group-major, so each bracket spans only its own readers.
• W2 — withdrawn rather than argued: with group-major bracketing each bracket really is one group's read section, so the cpu_l3/cpu_dtlb ruling applies for the reason it applies there. Comment rewritten to say condition (c) is met by construction, not asserted.
• W3 — a bracket that wrote nothing now discard()s instead of finish()ing.
• W4 — fixed by the per-metric split above.

core_cstate_residency keeps a group whose members are the union of the core levels — the one place a union is right, since every core-scope reader writes it.

You were also right that Reader.acq's doc comment described behaviour refresh didn't have. It does now, and the comment says why group-major costs nothing: independent perf fds, no per-device handle, no re-visit to trade against.

Dashboard scaling

The engine has no and/unless or group_left, so the aggregate uses the join that already works: avg(sum by (id)(…) / sum by (id)(cpu_tsc)), whose keyed join drops logical CPUs with no core-scope counterpart — the same thing that made the sum by (id) companions correct and hid the bug.

Package plots divide by avg(irate(cpu_tsc)), not sum: a package counter ticks at one package's TSC rate, so the matching whole is one CPU's worth. Flagging the assumption explicitly — this leans on TSC being invariant and equal-rate across CPUs. Happy to key it to a package-representative CPU instead if you'd rather not rely on that.

Smaller items

All five taken. PSYS opens only the first cpumask entry (kept Index::Zero — it is host-scope; the bug was opening one reader per package for one platform-wide quantity). Index::resolve returns Option and is called before open_counter, so an over-capacity index costs no fd at all, not just no syscall. DOMAIN_ALIASES gains ("core","cpu") and ("package","cpu") — checked that no sampler name in EXPECTED_SUBSYSTEMS starts with core_/package_, so applying aliases to sampler names stays harmless. Lone plots use plot_promql_full. Unit::Energy and the energy unit system are removed rather than kept unused, and the stale cpu_*_power gauge references are corrected in both the module header and cpu.rs:361.

Verification

Ran the sampler on sky (12t/6c), hyb (16t/10c) and amd (24t) through systemslab, staged sysbench idle→1→half→all→idle, turbostat alongside.

Membership — exact match on all three, recorded metric set vs PMU-exposed set checked programmatically against sysfs. amd is the W4 case: its power PMU exposes energy-pkg alone, and cpu_cores_energy/cpu_igpu_energy/cpu_dram_energy now come back Metric not found rather than as flat zeros. In the viewer, amd's CPU dashboard shows one Power subgroup and no C-State group at all instead of empty subgroups.

Residency scaling — matches your arithmetic:

host	fixed	pre-fix	you predicted
sky (12t/6c)	0.989	0.495	0.50
hyb (16t/10c)	0.995	0.622	0.625

Power vs turbostat — amd 72.2 W vs 71.1 W; hyb traces 3.7 W idle → 50 W all-cores → 3.8 W.

One thing I could not demonstrate. The package-residency sum→avg change is correct by construction but has no measurement behind it: every package C-state counter read flat zero across both Intel hosts for the whole run. These parts don't enter a package idle state while an agent, a recorder and turbostat are all running. The membership assertions still cover which package levels exist; only the scaling is unverified. If you have a host that idles deeply enough, that's the gap.

[brayniac]

Re-reviewed in a worktree with the branch merged onto current main (6c014d3d, ~30 commits past its merge base). Two independent passes — a manual read and a /code-review high agent — converged on the same top two findings.

Merge/build state: merges cleanly. cargo check --workspace --all-targets clean; cargo test --bins 689 pass, cargo test -p dashboard 80 pass, analysis/extract 91 pass; scripts/check-viewer-symlinks.sh resolves. The Linux-only sampler body is not compiled on the review host (macOS).

Last round's two blocking findings are fixed, correctly. set_member_set replaced the prefix bound, and the split went past what I asked for — one group per metric, which is the right granularity for exactly the reason given: snapshot.rs applies member_set() per metric, not per group.

Three things below; the first two are the ones I'd want changed before merge.

1. A host with no RAPL PMU is reported as disabled, not unsupported

src/agent/samplers/cpu/linux/power/mod.rs:180

Ok(None) lands in set_disabled() (src/agent/mod.rs:150) — the same state enabled = false produces. So on every VM, which is the case the comment right above it names, /samplers and rezolus status will report cpu_power as turned off by config.

SamplerState::Unsupported's own doc comment rules this out in as many words:

Deliberately not Disabled either: nobody turned it off, and an operator reading disabled would reasonably go looking for the config that did.

The established pattern for hardware absence is Err(sampler_status::Unsupported(...)) — l3/mod.rs:276 (the exact analogue: "no L3 cache domains"), tlb_flush/mod.rs:81, drivehealth/linux/mod.rs:92. This branch already contains the commits that drew that line (#1102 / #1103). docs/metrics.md carries the wrong word through too ("where the sampler disables itself").

2. core_cstate_residency is permanently windowless

src/agent/samplers/cpu/linux/power/mod.rs:243

CPU_POWER_CORE_CSTATE_ACQ is registered and gets a member set, but nothing ever brackets it. open_cstate throws the group away (total: total.map(|(metric, _)| metric), line 569), and refresh iterates self.groups, built from an ACQ_GROUPS list that deliberately excludes it. So no acquire()/finish() ever touches its slot.

The comment at line 232 says its window "is not stamped separately: the readers that feed it are bracketed by their own level groups." That is true of the values, but a level group's bracket stamps that level's own GroupWindowSlot, not this one. Concretely: AcquisitionGroup::window() returns None for cpu_power/cpu_power_core_cstate for the life of the process, so create_v3 emits it as GroupSnapshot { window: None } (snapshot.rs:1694), and the .rez table for core_cstate_residency carries null :window_begin/:window_width for the whole recording.

Consequence: no acquisition-window uncertainty band on rate()/irate() for that metric in the viewer or mcp query, while all eight per-level siblings beside it have one — and it is the metric behind the dashboard's headline "Total C-State Residency" plot. Rows are still written, so this is silent degradation rather than data loss. It is also the operational-checklist item verbatim: are the member metrics tagged acq_group 1:1 with the builder wiring?

Fix is small: one outer CORE_CSTATE bracket around the core-cstate portion of the sweep. The union of the level reads is the honest window for a summed metric.

3. The sweep is serial, where both nearest precedents are parallel

src/agent/samplers/cpu/linux/power/mod.rs:289

PowerInner::refresh() issues every Counter::read() inline on the async worker. The two structurally identical perf samplers in this tree — cpu_frequency (frequency/mod.rs:94) and cpu_l3 (l3/mod.rs:91) — both fan their reads out to dedicated per-core threads and join_all them inside the acquisition bracket.

That difference already shows in this PR's own numbers: cpu_frequency measured lower (316 / 585 us) than cpu_power (1002 us) on the same hosts, despite comparable counter counts.

It matters because ~11-27 us per counter is high for a read(2), and the reason is structural: a perf event opened with .one_cpu(N) and read from another CPU goes through smp_call_function_single — a cross-CPU IPI per counter, which costs the reading thread and perturbs the core being measured. cstate_core opens one counter per physical core per level, so a 2x32-core server exposing c1+c6 is ~140 serialized reads and ~140 IPIs per refresh, against a default snapshot ttl of 10ms, with the sampler on by default (only gpu_amd_pmu is opt-in).

The hosts measured are 12t / 16t / 24t. The reviewing-samplers checklist asks for the number at max CPU count rather than the dev box, and the PR already lists multi-socket as untested. Either match the thread pattern, or post the high-core-count measurement so the serial sweep is a known quantity.

Smaller

• src/viewer/assets/lib/features/explorers.js:6 — UNIT_OPTIONS was not extended with power. Line 333 seeds unitOverride from plot.opts.format.unit_system, so opening the plot explorer on any of the six new Power plots selects a value the <select> has no <option> for: it displays "Auto (none)", and the first touch of the dropdown silently drops the watt formatting. Shared frontend module, so it affects both viewers. Add { value: 'power', label: 'Power (W)' }.
• units.js power system has no sub-watt scale. It starts at W with precision 2, so the idle-iGPU microwatt domain — the case cited to justify deriving power from energy rather than the milliwatt gauge — still renders 0 W. mW / uW steps would close that.
• power/mod.rs:493 — a whole energy domain can drop silently. event_scale() returning None bails with no log, and Dynamic::builder(pmu).ok()? is silent in both helpers. scale() returns Ok(None) when the .scale sidecar is merely absent, so an event the PMU does expose but whose sysfs lacks .scale disappears with zero diagnostic — while every other failure path in the file emits a debug!. A debug! here makes "why is cpu_dram_energy missing on this box?" answerable from RUST_LOG=debug.
• power/mod.rs:472 — warn amplification. Index::resolve warns per (pmu, event, cpu) triple. Index::Cpu is used across every core-scope cpumask entry and every exposed level, so a host with core ids past MAX_CPUS = 1024 emits one line per level per out-of-range core. A once-per-(pmu, event) summary ("N of M CPUs beyond the 1024-entry ceiling") carries the same information.
• No unit tests. Index::resolve is pure, and it covers exactly the multi-socket ordinal path the PR flags as untested — testable without the hardware.

Checked and correct, so not listed above

The microjoule carry-remainder accumulator (no truncation drift); the counter-reset / checked_sub re-baseline; std::ptr::eq group partitioning (every AcquisitionGroup is a static, so address identity is sound); the core_cstate_residency member-set union; Index::Zero's single-reader rule for energy-psys; PmuKind::Rapl being unbudgeted (plan() grants every non-Core kind); the DOMAIN_ALIASES additions; and the new PromQL — avg(<binary>) is supported, and sum by (id)(x) / avg(y) matches via matrix_matrix_op's single-right broadcast fallback, so the package residency plots do work despite the label-set mismatch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KheDzaD5bnVWcmdocmk2eW

[yangxi]

Thanks — all three of the top findings were right, and the second one was wrong rather than merely wide, exactly as you said. Two are fixed; the third I'm pushing back on, with measurements. Pushed as a2d2253.

1. disabled → unsupported (fixed)

You're right that Ok(None) lands in set_disabled() and that a VM would report cpu_power as turned off by config. Now returns Err(sampler_status::Unsupported(...)), the same call cpu_l3 makes for a part with no L3 domains. docs/metrics.md carried the wrong word through too; fixed there as well.

2. core_cstate_residency was permanently windowless (fixed, and confirmed on hardware)

Exactly as described: CPU_POWER_CORE_CSTATE_ACQ got a member set but nothing ever bracketed it, so window() returned None for the life of the process.

refresh() now opens one outer bracket around the whole sweep and stamps it if any core-scope level wrote — the union of the level reads, which is the honest span for a summed metric. It discard()s when nothing wrote, same rule as the per-group brackets.

Verified by reading the :window_width column straight out of a .rez, before and after, on hyb:

	rows	stamped	null
pre-fix (hyb)	21	0	21
post-fix (hyb)	161	161	0
post-fix (sky)	161	161	0

Post-fix window width p50 is 775 us on hyb, 426 us on sky.

One note worth recording, because it nearly cost me the regression test: I first tried asserting on mcp query's uncertainty band, and that band is present even when every window is null — the query engine falls back to a sampler-level window for a windowless V3 table. It would have passed both before and after the fix. The null count in the column is the ground truth; the band is not.

3. The serial sweep — measured four ways, and I'd like to keep it

This is the one I'm disagreeing with, so here is the data rather than an argument.

I built the two structural alternatives plus a third, selectable at runtime so every arm runs the same binary, and measured n≈619 per cell on all three hosts, idle and under full sysbench load. The instrument is cpu_power sampling latency from refresh_with_logging.

• serial — current code, reads inline on the async worker.
• thread1 — one unpinned worker thread. This is what cpu_frequency/cpu_l3 actually do on bare metal: spawn_threads() only picks the pinned per-core path when is_virt().
• pin0 — one worker pinned to cpu0, where every package-scope counter is opened.
• percpu — one worker per CPU, each pinned to the CPU its counters were opened on. The only arm that removes the IPI rather than relocating it.

Under load — serial wins on every host

host	serial	thread1	pin0	percpu
hyb (38 ctr)	52 us	65 (+25%)	59 (+14%)	64 (+23%)
sky (28 ctr)	52 us	71 (+37%)	70 (+35%)	82 (+58%)
amd (13 ctr)	24 us	33 (+38%)	35 (+46%)	38 (+58%)

p99 is worse than the median: on amd percpu is 79 us against serial's 26 (+204%).

Idle — percpu wins big

host	serial	percpu
hyb	1002 us	241 us (−76%)
sky	~794 us	522 us (−34%)
amd	44 us	36 us (−18%)

Why the two regimes disagree

The cost is C-state exit latency, not IPI compute. hyb's deepest idle state has a 1048 us exit latency and its serial idle p50 is 1002 us — each cross-core read wakes a sleeping core. Under load the target core is already awake, so a read costs ~1.4–1.9 us and that rate is flat across all three hosts; what's left is pure handshake overhead.

So: idle cost = exit latency × remote counters (pinning removes it); busy cost = ~1.6 us × counters (nothing to remove, threading only adds).

Two things fall out of this that I think matter more than the specific numbers:

Copying the cpu_frequency pattern would have accomplished nothing. thread1 — same single unpinned thread that sampler uses on bare metal — moved hyb's idle p50 by −0.5% to +0.0%. It relocates the IPIs off the async worker without eliminating one.

Idle is the expensive case, ~19× busy on hyb (1002 vs 52 us). Measuring a perf sampler under load, which is the intuitive test, hides almost the whole cost. That seems worth putting in the reviewing-samplers checklist independently of this PR.

Extrapolating serial's flat ~1.6 us/counter to a 2×32-core server gives roughly 320 us at ~200 counters under load. Real, but not alarming against a 10 ms ttl — and percpu would add ~64 handshakes on top of reads that were already cheap.

Still unmeasured: multi-socket. All three hosts are single-socket desktop parts, so I can offer counter-count and C-state scaling across three points, not the 2×32-core number you asked for. Flagging that rather than extrapolating past it.

Also unmeasured: partial load. My busy condition pegs every thread, which is the most favourable case for serial since no core is ever idle. A half-loaded machine sits between the two regimes and is arguably the most realistic fleet state. Happy to measure that if you'd rather decide on it.

Smaller items — all five taken

• explorers.js UNIT_OPTIONS — added { value: 'power', label: 'Power (W)' }.
• units.js sub-watt — added uW/mW steps below W. The idle-iGPU microwatt case now renders as 500 uW instead of 0 W; checked across ten orders of magnitude. Kept threshold: 0 on the lowest scale, since the selector falls back to scales[0].
• event_scale silent drop — all three failure paths now debug!, including the Ok(None) case where the .scale sidecar is merely absent, which was the one that could drop a whole domain with no diagnostic.
• Warn amplification — Index::resolve returns the rejection; callers count skips and emit one warn! per (pmu, event) naming the count and the ceiling, instead of one line per level per core.
• Unit tests — five on Index::resolve, covering exactly the multi-socket ordinal path this PR flags as untested: that a 0,64 cpumask becomes ordinals 0 and 1, that core scope indexes by cpu id instead, that Zero collapses, and that an over-capacity index is declined rather than clamped.

Verification

cargo fmt --check clean, cargo clippy --bins zero warnings, cargo test --bins 749 passed, node --test tests/*.mjs 242 passed (wasm_viewer_histogram_kpis still needs a prebuilt site/viewer/pkg/, fails identically on a clean tree), check-viewer-symlinks.sh 57 modules resolve.

Re-validated on all three hosts under the staged sysbench profile — VERDICT: pass on sky, amd and hyb. Each exercises a different shape, which is what makes the membership assertions worth anything: amd has power_core/energy-core and no cstate PMUs at all, sky exposes seven package C-state levels, hyb has three core levels behind an SMT-sparse cpumask (0,2,4,6,8,10,12-15). Every present/absent check matched the PMU's own event list.

Two incidental confirmations from those runs:

• sky's cpu_igpu_energy measures 15.7 uJ/s ≈ 16 uW — a live instance of the microwatt domain that motivated the sub-watt scales above. It renders 16 uW now instead of 0 W.
• hyb's cpu_igpu_energy is exactly zero for a whole run: it has a discrete RTX 5080, so the integrated GPU is powered down and the counter never advances. Worth knowing that a zero there is a hardware fact rather than a fabricated one — the membership assertions, which check against the PMU event list, are what catch the fabricated case.