Reserved address space is not a hole in the walk's coverage

CHANGELOG.md

@@ -6,8 +6,57 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+### Added
+
+- **`DebugEngine::virtual_region`** — `IDebugDataSpaces2::QueryVirtual` as a typed answer
+  (`VirtualRegion`, `VirtualState`), which is what the memory manager says about a run of pages
+  rather than what the debugger can read there. `VirtualState::Unknown(u32)` keeps a state this
+  crate does not name instead of folding it into reserved or committed, because either guess is
+  what would make the primitive lie. User-mode only; a kernel session has no answer to give and
+  says so as an error.
+
+  **Its output buffer must be 16-byte aligned**, which `MEMORY_BASIC_INFORMATION64` is not: the
+  engine's live-target path copies the 48-byte answer out with three `movaps` stores and takes
+  an access violation *inside dbgeng* otherwise. The dump path copies field by field, so the
+  same call against a full dump answered 22 queries from an 8-aligned buffer without complaint —
+  testing this on a dump proves nothing about it. Measured on 26200, 2026-09-24
+  (`dbgeng!Ordinal367+0x14f96`, `movaps xmmword ptr [rbx],xmm0`).
+
+- **`examples/heap_coverage.rs`** — what, if anything, holds a user heap walk short of
+  `Complete`: every gap it filed, put back to the memory manager, with an allocated chunk and a
+  free one as controls. An address as a second argument answers that one question, which is how
+  the dump direction is checked.
+
 ### Changed
 
+- **A heap walk no longer calls reserved address space a hole in its own coverage.** A user-mode
+  walk came back `coverage: Partial` on every healthy live process, because the tails of
+  subsegments and page ranges — reserved and never committed — read the same way a paged-out
+  page does, and *would not read* was the only thing the walk could observe. `PoolState` gains
+  `Uncommitted` (and `HeapState` with it), and `PoolState::is_coverage_gap` is now the single
+  definition of which gap costs a walk its `complete`.
+
+  What separates them is `virtual_region`, not the allocator's records: `CommittedPageCount`,
+  `CommitBitmap` and the LFH commit state are three structures that move between builds and say
+  what one allocator believes, while the memory manager answers about the target in one call.
+  Only a positive `MEM_RESERVE`/`MEM_FREE` excuses a gap — a failed query, a run that cannot
+  advance, an unnamed state and a source that cannot be asked are each conservative, so the
+  excuse is never granted by an absence of evidence. The kernel pool walk is not asked at all
+  and is unchanged.
+
+  A free chunk whose middle the allocator decommitted is the same question reached another way:
+  it runs past the committed extent it starts in, and `walk_vs` emitted no span for it and
+  cleared `complete` **without a diagnostic**. A span is geometry and state, both of which are
+  known there, so where the tail holds nothing the chunk is now reported; where it is memory the
+  process has, it is still refused, and the walk now names the chunk it dropped.
+
+  Measured on a live 26200 process (`sihost`, four Segment Heaps, 20,426 chunks, 2026-09-24):
+  every one of its 33 gaps `MEM_RESERVE`, both controls `MEM_COMMIT`, and an answer that had
+  been `Partial` is `Complete`. Checked from the other end on a thin dump of the same process:
+  an address whose page the dump does not carry still answers `Committed`, the read still fails,
+  and the walk still counts it. `HeapWalkReport` gains `uncommitted_gaps` so that what a
+  `Complete` answer forgave is still on the report. windbg-mcp `FOLLOWUPS.md` item 98.
+
 - **The kernel pool walker takes ARM64 targets.** `pool::query` accepted
   `IMAGE_FILE_MACHINE_AMD64` and nothing else, so every `pool_*` query against an ARM64 kernel
   came back `pool walking supports x64 targets only (machine 0xaa64)`. It now admits ARM64

Cargo.toml

@@ -63,6 +63,10 @@ features = [
     "Win32_Foundation",
     "Win32_System_Diagnostics_Debug",
     "Win32_System_Diagnostics_Debug_Extensions",
+    # `MEMORY_BASIC_INFORMATION64` and the `MEM_*` state constants, which is what
+    # `IDebugDataSpaces2::QueryVirtual` answers in — the memory manager's own record of which
+    # pages are committed, as `DebugEngine::virtual_region` returns it.
+    "Win32_System_Memory",
     "Win32_System_SystemInformation",
     # `RtlUpcaseUnicodeChar`, which is how `object::same_object_name` folds a name: it is the
     # object manager's own fold rather than a reproduction of it. Under `Wdk` rather than `Win32`
@@ -81,11 +85,16 @@ disarm64 = "0.2"
 
 # Only `examples/user_heap_smoke.rs` allocates on the host heap it then walks. Cargo unifies
 # this with the normal dependency when building examples and tests, and a downstream consumer
-# of the library still resolves the four features above and no more.
+# of the library still resolves the features above and no more.
+#
+# `Win32_System_Memory` is deliberately **not** repeated here even though the example calls
+# `HeapCreate`/`HeapAlloc` from it: the library dependency now carries it, and listing it twice
+# would leave a reader unable to tell which of the two is load-bearing — so deleting it from the
+# library list above would still build, and the deletion would only be found by a downstream
+# consumer.
 [dev-dependencies.windows]
 version = "0.62.2"
 features = [
-    "Win32_System_Memory",
     "Win32_System_Threading",
     # `GetConsoleProcessList` and `EnumWindows`/`IsWindowVisible`, which is how the launch tests
     # check that a debuggee gets a console of its own and that the console has no window

README.md

@@ -305,6 +305,7 @@ Per-session paged heaps are outside the initial pool-map scope, and the command
 | `examples/split_open.rs` | Re-validates the two-step openers, including a guard dropped before the engine is pumped. |
 | `examples/typed_context.rs` | Typed reads next to the debugger's own text for the same state. |
 | `examples/user_heap_smoke.rs` | Launches a child, allocates across size regimes, walks its Segment Heap and checks it against the child's own `HeapWalk`. |
+| `examples/heap_coverage.rs` | What, if anything, holds a user heap walk short of `Complete` — every gap it filed, against the memory manager's own record of what is behind it. |
 | `examples/register_description.rs` | The full register description, not one flag of it. |
 
 ## Building

docs/unknown-not-absent.md

@@ -65,10 +65,62 @@ Two further properties of it are load-bearing:
 `WalkCoverage`. No caller has to know that "incomplete" has more than one cause, and none can
 invent the distinction differently.
 
-**It is not implied by the diagnostics.** A walk can end incomplete having said nothing at all —
-`walk_vs` clears completeness when a readable region stops mid-chunk, without a message. A
-caller that wants to reject partial results consults `coverage` and never the message list.
-The inverse also holds: a walk with thousands of diagnostics can be `Complete`.
+**It is not implied by the diagnostics.** A walk can end incomplete having said nothing at all,
+and a walk with thousands of diagnostics can be `Complete`. A caller that wants to reject
+partial results consults `coverage` and never the message list. (`walk_vs` clearing
+completeness at a chunk running past a committed extent used to be the standing example of the
+silent case; it now names the chunk it dropped — see below.)
+
+### A page that will not read is three things, and only one of them is a gap
+
+A user-mode heap walk was `Partial` on every healthy live process. That is the same failure as
+reporting a partial reading as a total one, reached from the other side: a signal that fires on
+everything says nothing. What held it there was not damage. It was the tails of subsegments and
+page ranges — address space the allocator reserved and never committed — filed as `Unreadable`,
+because *would not read* was the only thing the walk could observe.
+
+Three states hide behind that one observation, and the failed read does not separate them:
+
+| What it is | What it means for coverage |
+|---|---|
+| Reserved, or decommitted — no pages behind it | Nothing could have been there, so nothing was missed. |
+| Committed and paged out, or absent from a dump | The target has this memory and the walk did not see it. A real gap. |
+| Committed and present | Not a gap at all — it read. |
+
+The allocator's own records can answer the first distinction: a page range descriptor's
+`CommittedPageCount`, a VS subsegment's `CommitBitmap`, an LFH subsegment's commit state at
+`CommitStateOffset`. That is three structures, each of which moves between builds, to learn
+what one allocator believes. The **memory manager** answers all of it in one call, about the
+target rather than about the allocator: `DebugEngine::virtual_region`, which is
+`IDebugDataSpaces2::QueryVirtual` returning `MEM_RESERVE`, `MEM_COMMIT` or `MEM_FREE`.
+
+So `PoolState::Uncommitted` joins `Unreadable`, and `PoolState::is_coverage_gap` is the single
+definition of which one costs a walk its `complete`. Three properties keep it honest:
+
+- **Only a positive answer excuses a gap.** A query that fails, a run that cannot advance, a
+  state this crate does not name, and a source that cannot be asked at all are each `None`, and
+  `None` keeps the conservative reading. The excuse is never granted by an absence of evidence.
+- **It is asked of the memory manager, never inferred from where the span lies.** Position is a
+  good guess and it is a guess: on a target trimming paged pool the pages that will not read
+  are committed and written, and a walk calling them empty would claim coverage it never had.
+- **A kernel session is not asked at all.** `QueryVirtual` is a user-mode question, so the
+  kernel pool walk keeps counting every unreadable page against its coverage — which, for paged
+  pool, is the truth.
+
+The same question answers a second one. A free chunk whose middle the allocator decommitted
+runs past the committed extent it starts in, and `walk_vs` emitted no span for it. A span is
+geometry and state, both of which are known there — the header was read, the size came out of
+it and passed the subsegment bound, the state comes from the free tree — and the only thing
+missing is the chunk's *contents*, which no span carries. So where the tail holds nothing the
+chunk is reported; where it is memory the process has, it is not, and now the walk says so.
+
+Measured on a live 26200 process (`sihost`, four Segment Heaps, 19,448 chunks, 2026-09-24): all
+48 gaps were `MEM_RESERVE`, both controls — an allocated chunk and a free one — were
+`MEM_COMMIT`, five chunks with decommitted middles were the last thing holding the walk short,
+and the answer that had been `Partial` was `Complete`. The dump direction was checked from the
+other end, on a thin dump of the same process: an address whose page the dump does not carry
+still answers `Committed`, the read of it still fails, and the walk still counts it. A dump
+that lacks committed pages is exactly the case this must not sweep up.
 
 ### Running out of time is not an error
 
@@ -415,6 +467,8 @@ Once you adopt it, it stops being a pool-walker concern.
 | `Module::name` empty | For an unloaded module there is no name to qualify symbols by. Empty is the fact, not a truncation. |
 | `PoolSpan::requested_size: Option<u64>` | Set only where allocator metadata validates it. Kernel pool and LFH/VS spans leave it `None` rather than guessing from capacity. |
 | `PoolState::Unreadable` | A distinct state from `Allocated` and the two free states — the walker reached the chunk and could not read it. |
+| `PoolState::Uncommitted` | Distinct from `Unreadable`, and the one gap that does *not* cost the walk its `complete`: the memory manager says there are no pages behind it, so nothing was missed. Never inferred — only a positive `MEM_RESERVE`/`MEM_FREE` puts a span here. |
+| `VirtualState::Unknown(u32)` | A `State` the engine returned that this crate does not name, kept rather than folded into reserved or committed — guessing either way is the one thing that would make the primitive lie. |
 | `query::chunk_at` → `Ok(None)` | "Not covered by the snapshot at all", which is a different answer from "it is a free hole" — that comes back as a chunk whose `PoolState` is not `Allocated`. |
 | `find_tag` indexes allocated chunks only | A freed chunk's tag is not reliably preserved by the allocator, so "freed chunks with this tag" would be inventing information. |
 | `PoolKind`'s eight variants | Not collapsed to paged/nonpaged, because crossing one of those boundaries creates false holes. |

examples/heap_coverage.rs

@@ -0,0 +1,159 @@
+//! Opt-in live probe: what, if anything, holds a user heap walk short of `Complete`.
+//!
+//! ```text
+//! cargo run --example heap_coverage -- 6176                 # a live process, by pid
+//! cargo run --example heap_coverage -- C:\dumps\sihost.dmp  # or a user-mode dump
+//! cargo run --example heap_coverage -- C:\dumps\sihost.dmp 0x1ec81102040
+//! ```
+//!
+//! Walks every Segment Heap in the target, then asks the **memory manager** about every gap the
+//! walk filed — which is the question `PoolState::Uncommitted` turns on, and the only way to
+//! tell a reserved subsegment tail from a page the process has and the debugger could not read.
+//! With an address as a second argument it answers that one question and nothing else, which is
+//! how the dump direction is checked: a page a thin dump does not carry still answers
+//! `Committed`, and reading it still fails.
+//!
+//! Nothing here is asserted, because nothing here is a property of this crate — it is a reading
+//! of whatever target it is pointed at. Set `_NT_SYMBOL_PATH`, or the public symbol server is
+//! used.
+
+use std::collections::BTreeMap;
+use std::time::Duration;
+
+use dbgscope::dbgeng::{DebugEngine, VirtualState};
+use dbgscope::heap::{self, HeapState, HeapWalk};
+
+/// The walk's budget. Generous: this is a probe, and a run that expires reports the budget
+/// rather than the target.
+const BUDGET: Duration = Duration::from_secs(120);
+
+fn open(target: &str) -> Result<DebugEngine, Box<dyn std::error::Error>> {
+    let engine = DebugEngine::new();
+    engine.set_symbol_path(&std::env::var("_NT_SYMBOL_PATH").unwrap_or_else(|_| {
+        "srv*C:\\ProgramData\\dbg\\sym*https://msdl.microsoft.com/download/symbols".into()
+    }))?;
+    match target.parse::<u32>() {
+        // Already includes the break-in wait.
+        Ok(pid) => engine.attach_process(pid)?,
+        Err(_) => {
+            engine.open_dump(target)?;
+            engine.wait_for_event(60_000)?;
+        }
+    }
+    // The heap walker needs `ntdll`'s private types, and a deferred module has none.
+    engine.execute_command(".reload /f ntdll.dll")?;
+    Ok(engine)
+}
+
+fn describe(state: VirtualState) -> String {
+    match state {
+        VirtualState::Committed => "committed".to_string(),
+        VirtualState::Reserved => "reserved".to_string(),
+        VirtualState::Free => "free".to_string(),
+        VirtualState::Unknown(state) => format!("unknown state {state:#x}"),
+    }
+}
+
+fn coverage(target: &str) -> Result<(), Box<dyn std::error::Error>> {
+    let engine = open(target)?;
+    let answer = heap::allocations(&engine, HeapWalk::refreshed().within(BUDGET))?;
+    println!(
+        "coverage {:?}: {} chunks, {} unreadable gaps, {} uncommitted gaps",
+        answer.walk.coverage,
+        answer.found.len(),
+        answer.walk.unreadable_gaps,
+        answer.walk.uncommitted_gaps
+    );
+    println!(
+        "  refused {} headers, {:#x} unplaced bytes, {:?}",
+        answer.walk.refused_headers, answer.walk.unplaced_bytes, answer.walk.stalls
+    );
+
+    // Every gap, against the memory manager — one row per answer, sized. A walk that reports
+    // `Complete` should have nothing but reserved runs here; anything committed is memory the
+    // target has and the walk did not see, and is what the coverage figure is about.
+    let mut tally: BTreeMap<String, (usize, u64)> = BTreeMap::new();
+    let mut note = |label: String, bytes: u64| {
+        let row = tally.entry(label).or_insert((0, 0));
+        row.0 += 1;
+        row.1 += bytes;
+    };
+    for gap in answer.found.iter().filter(|gap| !gap.state.is_chunk()) {
+        let mut cursor = gap.header_address;
+        let end = gap.end();
+        while cursor < end {
+            match engine.virtual_region(cursor) {
+                Ok(region) if region.contains(cursor) => {
+                    let stop = region.end().unwrap_or(end).min(end).max(cursor + 1);
+                    note(
+                        format!("{:?} / {}", gap.state, describe(region.state)),
+                        stop - cursor,
+                    );
+                    cursor = stop;
+                }
+                Ok(region) => {
+                    note(
+                        format!(
+                            "{:?} / answered about {:#x}+{:#x}",
+                            gap.state, region.base, region.size
+                        ),
+                        end - cursor,
+                    );
+                    break;
+                }
+                Err(why) => {
+                    note(format!("{:?} / no answer: {why}", gap.state), end - cursor);
+                    break;
+                }
+            }
+        }
+    }
+    for (label, (runs, bytes)) in &tally {
+        println!("  {label}: {runs} runs, {bytes:#x} bytes");
+    }
+
+    // Controls. A chunk the walk *did* read has to come back committed; if it does not, the
+    // rows above are measuring the query rather than the target.
+    for state in [HeapState::Allocated, HeapState::ReusableFree] {
+        if let Some(chunk) = answer.found.iter().find(|chunk| chunk.state == state) {
+            println!(
+                "  control {state:?} {:#x}: {}",
+                chunk.user_address,
+                engine.virtual_region(chunk.user_address).map_or_else(
+                    |why| format!("no answer: {why}"),
+                    |region| describe(region.state)
+                )
+            );
+        }
+    }
+
+    let diagnostics = heap::diagnostics(&engine, HeapWalk::cached())?;
+    for shape in &diagnostics.found.categories {
+        println!("  {} x {}", shape.total, shape.shape);
+    }
+    engine.end_session()?;
+    Ok(())
+}
+
+fn one_address(target: &str, address: u64) -> Result<(), Box<dyn std::error::Error>> {
+    let engine = open(target)?;
+    println!("{address:#x}: {:?}", engine.virtual_region(address));
+    println!(
+        "  read: {:?}",
+        engine.read_memory(address, 8).map(|bytes| bytes.len())
+    );
+    engine.end_session()?;
+    Ok(())
+}
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let arguments: Vec<String> = std::env::args().skip(1).collect();
+    match arguments.as_slice() {
+        [target] => coverage(target),
+        [target, address] => one_address(
+            target,
+            u64::from_str_radix(address.trim_start_matches("0x"), 16)?,
+        ),
+        _ => Err("usage: heap_coverage <pid|dump> [address]".into()),
+    }
+}