[matt-welch]

Description

Two cases in test/t/23_fiction_worker_convergence.pl assert on short windows by polling, and both fail intermittently in CI on main since the search-slot change in #103. Case 3 is the TAP failure currently showing up on open PRs. This PR makes both cases deterministic by holding the system at the moment each one checks, using two new injection points. What each case asserts is unchanged.

• Case 3 checks that a database that isn't live yet is granted 0 search threads and parks 0 slots.
  • The 0 grant lasts only from the reconcile that spawns the worker until the worker's own launcher kick, a few milliseconds. On a loaded runner the poll loop usually saw only the stale grant left on the paused row, then the final grant of 5.
  • The new injection point vamana-worker-before-publish-pid in VamanaWorkerServe holds the spawned worker just before it publishes its pid. The case then reads the grant and the worker's search_slots_registered.
  • The slot check uses search_slots_registered instead of counting search slots in pg_stat_activity. A slot parked before the worker publishes its pid fails its first owner check and exits right away, so the activity count reads 0 whether or not a slot was parked.
• Case 13 checks that a crashed worker's search slots outlive it when its release-on-exit callback fails, then exit on their own.
  • The launcher's dead-entry cleanup clears workerPid. After that, an orphan can reach its next owner poll and exit before the test samples the survivors.
  • The new injection point vamana-launcher-before-reconcile in the launcher loop holds the launcher across the crash. The crash points are detached before the launcher is released, so the respawned worker doesn't trip them a second time.
• INJECTION_POINT compiles to nothing on builds without injection points. On those builds the two case 3 assertions are skipped, as cases 12 and 13 already are.

No product behavior changes.

Related Issues

Type of Change

• Bug fix
• New feature
• Refactor / code cleanup
• Documentation update
• Test addition or update
• Build / CI change

Pre-Merge Checklist

Build

• make completes without errors or warnings
• make install completes successfully

Tests

• If this PR introduces no new behavior: existing regression tests (make installcheck) and TAP tests (test/t/) pass with no failures
• If this PR introduces new behavior: test cases covering it were added to test/sql/ and/or test/t/
• If this PR adds a standalone unit-test module under test/modules/: it builds and passes (make -C test/modules/<module> installcheck)

Documentation

• Relevant docs under docs/ updated if architecture or usage changed

Testing Notes

All runs were on this branch, which is based on current main, with injection points enabled.

• Build: make WERROR=1 and make install are clean, with no warnings.

• Full suites:

  • Full TAP suite: 72 files, 1529 tests, all pass.
  • SQL regression with ci/test_server.conf: 10/10.
  • Hardware tests: prove_installcheck-hw passes (2 files, 55 tests), and installcheck-hw with ci/test_server.conf passes 2/2.

• Flake rate: I ran 23_fiction_worker_convergence.pl 25 times under CPU contention: pinned to two CPUs with taskset, plus a busy loop on each CPU.

  Tree	Failing runs
  main before Skip search slots until a database has an index #103	2/25 (case 13)
  current main, without this change	21/25
  this change	0/25

• The new assertions can still fail: each was checked against an injected regression.

  • Clamping the worker's slot count to at least 1 fails the case 3 slot assertion.
  • Granting threads to a database that isn't live fails the case 3 grant assertion.
  • Running release-on-exit before the crash point fails the case 13 survivor assertion.

Other intermittent failures seen on main recently (58_residency_rollback_reconcile.pl, 73_primary_slot_catchup_scale.pl, and one vamana_databases SQL failure) are not addressed here. 58 and 73 predate #103.