Skip to content

Forward load errors; document format upgrades - #89

Open
matt-welch wants to merge 3 commits into
mainfrom
fix-ondisk-format-upgrade-docs
Open

matt-welch wants to merge 3 commits into
mainfrom
fix-ondisk-format-upgrade-docs

Conversation

@matt-welch

Copy link
Copy Markdown
Contributor

Description

When a Vamana index fails to load, the worker-mediated search path showed clients only a generic "index not loaded" error with SQLSTATE XX000, even though the worker had already logged the real cause to the server log. INSERT and VACUUM, which check the index directly on the backend, always surfaced the real error with a hint; only SELECT lost it on the way through the worker.

VamanaWorkerGetOrLoadIndex in src/vamanaworkerindex.c now records the sqlerrcode, message, and hint of a load failure right before it is logged and freed, in worker-local state (the worker is single-threaded, so only one load is ever outstanding). A new VamanaWorkerGetLastLoadError() accessor exposes it. The generic "not loaded" error in src/vamanaworkersearch.c, raised when a load attempt returns nothing usable, now forwards that real error (as SQLSTATE 22000, data exception) instead of a message with no connection to the actual cause.

test/t/01_persistence.pl had an existing test for a non-finite vector value forcing a failed index rebuild; it asserted the old generic client-facing message, which this change replaces with the real one, so that assertion was updated.

Added test/t/68_vamana_metapage_format_upgrade.pl: it patches a real index's on-disk magic number back to the value used before the on-disk format changed, then drives it through INSERT, VACUUM, SELECT, REINDEX, REINDEX CONCURRENTLY, and DROP INDEX, confirming each either fails closed with the correct error and hint or restores service, and that the worker never crashes.

docs/USER_GUIDE.md gained an "Upgrading the Extension" section explaining that an on-disk format change requires rebuilding affected indexes with REINDEX, since there is no ALTER EXTENSION ... UPDATE path that can rewrite index pages, plus a REINDEX trigger bullet, a troubleshooting entry for the error text, and a cross-link from the existing halfvec upgrade note.

Related Issues

Type of Change

  • Bug fix
  • New feature
  • Refactor / code cleanup
  • Documentation update
  • Test addition or update
  • Build / CI change

Pre-Merge Checklist

Build

  • make completes without errors or warnings
  • make install completes successfully

Tests

  • If this PR introduces no new behavior: existing regression tests (make installcheck) and TAP tests (test/t/) pass with no failures
  • If this PR introduces new behavior: test cases covering it were added to test/sql/ and/or test/t/
  • If this PR adds a standalone unit-test module under test/modules/: it builds and passes (make -C test/modules/<module> installcheck)

Documentation

  • Relevant docs under docs/ updated if architecture or usage changed

Testing Notes

Full TAP suite run via the project's build-and-test tooling: 61 files, 1386 tests, all passing, including the new test/t/68_vamana_metapage_format_upgrade.pl (25 assertions) and the updated test/t/01_persistence.pl. SQL regression suite run separately: 8 files, all passing.

The error-forwarding change was verified by reverting it (git apply -R on just that part of the diff), confirming the three assertions tied to it fail for the expected reason, then reapplying and confirming they pass again.

The new TAP test's own corruption mechanism was verified the same way: with the magic-number patch made a no-op, the INSERT and VACUUM failure assertions correctly report failure, confirming the test actually exercises the failure path rather than passing regardless.

A search against a Vamana index that fails to load (stale on-disk
format, a non-finite vector value forcing a failed rebuild, etc.)
previously told the client only a generic "index not loaded" error with
SQLSTATE XX000, even though the worker had already logged the real
cause. INSERT and VACUUM, which check the index directly on the
backend, always showed the real error; only the worker-mediated SELECT
path lost it.

vamanaworkerindex.c's VamanaWorkerGetOrLoadIndex now records the
sqlerrcode, message, and hint of a load failure it is about to log,
right before freeing it, in worker-local static state (the worker is
single-threaded, so only one load is ever in flight). A new
VamanaWorkerGetLastLoadError() accessor exposes it.
vamanaworkersearch.c's generic "not loaded" error, raised when a load
attempt comes back empty-handed, now forwards that real error (as
SQLSTATE 22000, data exception) instead of synthesizing a message with
no connection to the actual cause.

test/t/01_persistence.pl's existing non-finite-vector-rebuild test
asserted the old generic message; updated it to expect the real one,
since it exercises the same code path.

Added test/t/68_vamana_metapage_format_upgrade.pl, which patches a real
index's on-disk magic number back to the pre-format-change value and
drives it through INSERT, VACUUM, SELECT, REINDEX, REINDEX CONCURRENTLY,
and DROP INDEX to confirm each one either fails closed with the right
error or restores service, with no worker crash.

docs/USER_GUIDE.md gained an "Upgrading the Extension" section
explaining that on-disk format changes require a REINDEX and are not
covered by an ALTER EXTENSION UPDATE script, plus a REINDEX trigger
bullet, a troubleshooting entry, and a cross-link from the existing
halfvec upgrade note.

Signed-off-by: Matt Welch <matt.welch@intel.com>
@matt-welch
matt-welch requested a review from a team October 14, 2026 18:45

@asonje asonje left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src/vamanaworkersearch.c:172 hardcodes slot->errorCategory = VAMANA_ERR_DATA; regardless of what the load actually failed with, even though loadErrSqlerrcode (VamanaWorkerGetLastLoadError's first out-param) is captured right above it and never used otherwise.

This project already has the right codec for this: VamanaCategorizeSQLState(int sqlerrcode) (src/vamanaworkershmem.c:1366-1376), used the same way at three existing call sites in vamanaworkerwrite.c (579, 772, 848). Swapping in slot->errorCategory = VamanaCategorizeSQLState(loadErrSqlerrcode); would make the client-visible SQLSTATE for a worker-mediated SELECT match the real failure (e.g. ERRCODE_OUT_OF_MEMORY/ERRCODE_IO_ERROR) instead of always reporting
ERRCODE_DATA_EXCEPTION, consistent with how the message/hint are already forwarded verbatim.

@matt-welch
matt-welch marked this pull request as draft October 16, 2026 01:29
@matt-welch
matt-welch marked this pull request as ready for review October 19, 2026 22:25
Resolves a conflict in docs/USER_GUIDE.md where this branch's new
"Upgrading the Extension" section and main's compact_threshold_pct
superuser note were both inserted at the same point; both are kept,
with the superuser note staying attached to its compact_threshold_pct
section ahead of the new section. Everything else from main's seven
newer commits merges in cleanly on its own.

Signed-off-by: Matt Welch <matt.welch@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants