fix(cluster): re-pull a node's own files after an empty-disk restore; hold the manifest sweep until then

[xe-nvdk]

Summary

Fixes #959, found during the #957 review. On per-node storage with file replication the puller assumed a node still holds every file it once wrote: enqueue returned SkippedSelf for any entry whose origin was this node, and the startup catch-up walk fast-pathed the same entries. A node restored with an empty data disk under a stable cluster.node_id (the StatefulSet shape) pulled every other node's files back and never its own, and its reads of those partitions returned fewer rows with no error, for good. An act-mode reconciliation on that node then found each of those entries missing locally and proposed its deletion, which since #958 every other node carries out on its replica.

• Presence decides, on per-node storage. New puller flag RepullMissingSelfOrigin, set by the coordinator when the backend is local. The startup and periodic walks check a self-origin entry inline (selfOriginPresent → statLocal + presentAtSize, the one presence rule the worker's pre-pull check now shares): present at the manifest's size → skipped as before with the same counters; missing, short, or un-stat-able → catch-up-tagged on the startup walk and pulled from a peer's replica like any foreign entry (the resolver already excludes this node as origin and fallback; FetchFile serves any manifest-listed file a peer holds; SHA verified). A short file is fully re-pulled. enqueue's self-origin fast path now applies to the reactive source only (this node just wrote the file) or when the flag is off. Inline rather than through the queue so a writer with many own files does one cheap stat per entry per walk with no queue or inflight churn and no change in what catchup_skipped_local/catchup_enqueued mean. Shared backends keep the old fast paths: the bundled enterprise-shared rig runs the puller on a shared bucket, where a missing own object is not on any peer either and the check would be one HEAD per own entry per walk.
• The manifest sweep waits for convergence. Reconcile checks ShouldRunManifestSweep before step 4 and, when withheld, sets run.ManifestSweepHeld (in the run's JSON, log and audit maps) and goes on to the storage half. The adversarial pass caught that routing a false gate through the sweep's chunk-boundary check instead would abort every run on a restoring node as lease_lost, skip the storage sweep and log an Error. In cmd/arc/main.go the local-storage gate returns coordinator.ReplicationReady() for the manifest half when replication and its catch-up walker are both enabled (the gate now takes a three-method interface so it is unit-tested), and warns once at startup when the walker is off, mirroring the query gate's rule, since readiness would never be reached.
• Strict predicate, by choice. FullyCaughtUp has a livelock: an own-origin file no peer holds any more keeps catchup_failed up and the node's manifest sweep held — for all its own orphans — until an operator clears it. The narrower predicate would re-open reconciliation: a node restored from an empty disk never re-pulls its own-origin files, and an act-mode run then proposes manifest deletes for them #959 whenever the only replica holder is down during the restore. Stated in the notes with the remedy.
• Release-notes entry under ## Bug fixes; docs in docs(arc-enterprise): catch-up re-pulls a node's own files; the manifest sweep waits for it docs.basekick.net#89 (clustering page: catch-up re-pulls own files, the manifest sweep waits, the walker-disabled callout).

Pre-existing and out of scope, named in the plan: with the walker disabled neither the re-pull nor the hold applies (Warn at startup; keep reconciliation in dry run after a restore); if the startup manifest barrier times out, own files replayed by Raft after that point are recovered by the next periodic pass; a puller that failed to start leaves ReplicationReady() true; a full-size .part left by a crash between write and rename reads as present to both checks and is never finalised (to file); the snapshot-install gap. Plan + matrix: docs/progress/2026-09-28-empty-disk-restore-own-origin-959.md (untracked).

Configuration matrix

Configuration	Reaches new code?	Preconditions established?
OSS / standalone	no — no coordinator, no puller, no reconciler	n/a
Cluster, replication_enabled=false	walks never run (no puller); the gate's hold is not armed	puller == nil snapshot in ReplicationReady
Shared bucket + replication on (the bundled enterprise-shared rig)	flag off (storage.Type() != "local"): walks keep the old fast paths; gate keeps IsActiveCompactor for both halves	c.storage set before Start
Pattern 1 + replication, stable id, empty-disk restore (the bug)	startup walk stats own entries, tags and enqueues the missing ones, pulls from peers; sweep held until caught up; run reports manifest_sweep_held=true	SelfNodeID = c.localNode.ID; resolver excludes self; catch-up tags own entries like foreign ones
Pattern 1 + replication, healthy node	one stat per own entry per walk, all present → skipped; counters unchanged	local stat
Pattern 1 + replication, generated node id	after a restart old entries are foreign → existing path	unchanged
cluster.replication_catchup_enabled=false (non-default)	no walks → no re-pull; gate does not consult readiness; Warn at startup	mirrors the query gate's rule
cluster.query_gate_on_catchup=true (non-default)	own missing files are catch-up-tagged → reads 503 until they are back	same predicate
reconciliation.enabled=true, dry run (default), restoring node	storage scan runs and reports; manifest_sweep_held=true; Aborted=false; no Error	pre-sweep hold
reconciliation.manifest_only_dry_run=false (non-default), restoring node	no manifest deletes proposed while held; none needed after catch-up	same
Own entry no peer holds any more	catchup_failed → gate red and manifest sweep held on this node until the operator remedy	same as an unpullable foreign entry
Only replica holder down during the restore	transient catchup_failed; the periodic walk heals it; the held sweep prevents the act-mode delete meanwhile	strict predicate
Restart replay / snapshot install	reactive own-origin registers skipped; the startup walk (after leader + barrier) covers them; barrier timeout → periodic walk	runCatchUpOnce ordering
Puller failed to start	ReplicationReady() true: no re-pull, no hold	Start continues without the puller
100k missing own files, few workers	walker paces at the queue high-water and holds the reconciliation mutex	existing backpressure
Stat error that is not ENOENT	not present → enqueue → processEntry classifies (ErrInvalidPath quarantine)	existing branch

Test plan

• go build ./cmd/... ./internal/..., gofmt -l empty, go vet on filereplication, reconciliation, cmd/arc
• go test -race ./internal/cluster/filereplication/ ./internal/reconciliation/ green (whole packages); go test -tags=duckdb_arrow ./cmd/arc/ -run TestReconciliationGate green
• New: TestCatchUpPullsMissingSelfOriginFile (pulled, catchup_enqueued=1, converges after), TestCatchUpPullsShortSelfOriginFile (a short .part is fully re-pulled), TestCatchUpSkipsPresentSelfOriginFile (skipped_self, catchup_skipped_local, no fetch, converged), TestReconciliationWalkPullsMissingSelfOriginFile (periodic walk pulls, then skips once present), TestCatchUpKeepsSelfOriginFastPathWithoutRepull (flag off keeps the old behaviour), TestReconcile_HeldManifestSweepStillRunsStorageSweep (held: Aborted=false, ManifestSweepHeld=true, ManifestDeletes=0, StorageDeletes=1), TestReconciliationGate_LocalHoldsManifestSweepUntilCaughtUp (six cases). Existing TestPullerSkipsSelfOrigin pins the reactive skip.
• Pre-fix proofs (revert-run-restore, one per part): with the walker's presence check forced to "present" (the old assumption) the three re-pull tests time out with nothing pulled; with the pre-sweep hold removed the held-sweep test fails with manifest sweep: reconciliation: gate revoked mid-run; with the gate's hold bypassed the "local, replicating, not caught up" case returns true
• Review: adversarial pass on the plan (its Blocker reshaped Part B: a false gate inside the sweep is the revocation path and aborts the run; its findings on the strict predicate, the shared-backend cost and the replay residual are folded in), then one deep reviewer with the matrix and the cluster-ops checklist (every row confirmed by trace; "the code itself is correct"). Its findings, all taken: H1 the gate test lives in the gitignored cmd/arc directory and had to be force-added; H2 the release note's livelock remedy was wrong and unsafe (with the walker off nothing is pulled and the sweep is not held, so an act-mode run deletes every own-origin entry still missing; the restart clears the in-process failure, not the delete) — rewritten to confirm from a dry run first, then two restarts; M1 the reactive exemption with the re-pull on was untested — TestPullerSkipsSelfOriginReactiveWithRepull added; M2/M3 comments that overstated the stat's bound or described the old self-check fixed; style: step-5 comment, the query gate is opt-in, the startup Warn fires only with reconciliation enabled, a nil-context guard in statLocal.
• Live (binary run, since cmd/arc/main.go changed): Pattern 1 rig built from this branch (enterprise-local: writer1-3 + reader1, replication on, catch-up on, tiering and reconciliation on every node, reconciliation in its default dry run). Six flushes written directly to writer2 so it originates them; six replicas on writer1, writer3 and reader1 within 2 s. Restore 1, data files wiped, Raft and WAL kept: writer2's catch-up completed 4 s after the start, the log shows six File pulled from peer lines for the wiped hour (peer arc-writer1:9100), the six files are back on disk, and a dry-run reconciliation on writer2 reports manifest_file_count=6 storage_file_count=6 orphan_manifest_count=0 aborted=false manifest_sweep_held=false. Restore 2, the whole volume wiped (Raft, WAL, SQLite, files, license cache): writer2 rejoined as a follower, replayed the log, pulled the same six files back (twelve pull lines for the hour over the two restores), and the dry run reports the same zeros. Before this change the same node came back with every other node's files and none of its own. Aside, pre-existing and unrelated: on both restarts the auth FSM logs Error for replayed token-create entries it refuses as duplicates (token name "..." already exists) — worth its own issue. The hold itself (manifest_sweep_held=true) was not observable live because the six pulls settle within the first seconds; it is covered by the held-sweep test.