Repository navigation
Forward load errors; document format upgrades - #89
matt-welch wants to merge 3 commits into
Conversation
A search against a Vamana index that fails to load (stale on-disk format, a non-finite vector value forcing a failed rebuild, etc.) previously told the client only a generic "index not loaded" error with SQLSTATE XX000, even though the worker had already logged the real cause. INSERT and VACUUM, which check the index directly on the backend, always showed the real error; only the worker-mediated SELECT path lost it. vamanaworkerindex.c's VamanaWorkerGetOrLoadIndex now records the sqlerrcode, message, and hint of a load failure it is about to log, right before freeing it, in worker-local static state (the worker is single-threaded, so only one load is ever in flight). A new VamanaWorkerGetLastLoadError() accessor exposes it. vamanaworkersearch.c's generic "not loaded" error, raised when a load attempt comes back empty-handed, now forwards that real error (as SQLSTATE 22000, data exception) instead of synthesizing a message with no connection to the actual cause. test/t/01_persistence.pl's existing non-finite-vector-rebuild test asserted the old generic message; updated it to expect the real one, since it exercises the same code path. Added test/t/68_vamana_metapage_format_upgrade.pl, which patches a real index's on-disk magic number back to the pre-format-change value and drives it through INSERT, VACUUM, SELECT, REINDEX, REINDEX CONCURRENTLY, and DROP INDEX to confirm each one either fails closed with the right error or restores service, with no worker crash. docs/USER_GUIDE.md gained an "Upgrading the Extension" section explaining that on-disk format changes require a REINDEX and are not covered by an ALTER EXTENSION UPDATE script, plus a REINDEX trigger bullet, a troubleshooting entry, and a cross-link from the existing halfvec upgrade note. Signed-off-by: Matt Welch <matt.welch@intel.com>
asonje
left a comment
There was a problem hiding this comment.
src/vamanaworkersearch.c:172 hardcodes slot->errorCategory = VAMANA_ERR_DATA; regardless of what the load actually failed with, even though loadErrSqlerrcode (VamanaWorkerGetLastLoadError's first out-param) is captured right above it and never used otherwise.
This project already has the right codec for this: VamanaCategorizeSQLState(int sqlerrcode) (src/vamanaworkershmem.c:1366-1376), used the same way at three existing call sites in vamanaworkerwrite.c (579, 772, 848). Swapping in slot->errorCategory = VamanaCategorizeSQLState(loadErrSqlerrcode); would make the client-visible SQLSTATE for a worker-mediated SELECT match the real failure (e.g. ERRCODE_OUT_OF_MEMORY/ERRCODE_IO_ERROR) instead of always reporting
ERRCODE_DATA_EXCEPTION, consistent with how the message/hint are already forwarded verbatim.
Resolves a conflict in docs/USER_GUIDE.md where this branch's new "Upgrading the Extension" section and main's compact_threshold_pct superuser note were both inserted at the same point; both are kept, with the superuser note staying attached to its compact_threshold_pct section ahead of the new section. Everything else from main's seven newer commits merges in cleanly on its own. Signed-off-by: Matt Welch <matt.welch@intel.com>
Description
When a Vamana index fails to load, the worker-mediated search path showed clients only a generic "index not loaded" error with SQLSTATE XX000, even though the worker had already logged the real cause to the server log. INSERT and VACUUM, which check the index directly on the backend, always surfaced the real error with a hint; only SELECT lost it on the way through the worker.
VamanaWorkerGetOrLoadIndexinsrc/vamanaworkerindex.cnow records the sqlerrcode, message, and hint of a load failure right before it is logged and freed, in worker-local state (the worker is single-threaded, so only one load is ever outstanding). A newVamanaWorkerGetLastLoadError()accessor exposes it. The generic "not loaded" error insrc/vamanaworkersearch.c, raised when a load attempt returns nothing usable, now forwards that real error (as SQLSTATE 22000, data exception) instead of a message with no connection to the actual cause.test/t/01_persistence.plhad an existing test for a non-finite vector value forcing a failed index rebuild; it asserted the old generic client-facing message, which this change replaces with the real one, so that assertion was updated.Added
test/t/68_vamana_metapage_format_upgrade.pl: it patches a real index's on-disk magic number back to the value used before the on-disk format changed, then drives it through INSERT, VACUUM, SELECT, REINDEX, REINDEX CONCURRENTLY, and DROP INDEX, confirming each either fails closed with the correct error and hint or restores service, and that the worker never crashes.docs/USER_GUIDE.mdgained an "Upgrading the Extension" section explaining that an on-disk format change requires rebuilding affected indexes with REINDEX, since there is noALTER EXTENSION ... UPDATEpath that can rewrite index pages, plus a REINDEX trigger bullet, a troubleshooting entry for the error text, and a cross-link from the existing halfvec upgrade note.Related Issues
Type of Change
Pre-Merge Checklist
Build
makecompletes without errors or warningsmake installcompletes successfullyTests
make installcheck) and TAP tests (test/t/) pass with no failurestest/sql/and/ortest/t/test/modules/: it builds and passes (make -C test/modules/<module> installcheck)Documentation
docs/updated if architecture or usage changedTesting Notes
Full TAP suite run via the project's build-and-test tooling: 61 files, 1386 tests, all passing, including the new
test/t/68_vamana_metapage_format_upgrade.pl(25 assertions) and the updatedtest/t/01_persistence.pl. SQL regression suite run separately: 8 files, all passing.The error-forwarding change was verified by reverting it (
git apply -Ron just that part of the diff), confirming the three assertions tied to it fail for the expected reason, then reapplying and confirming they pass again.The new TAP test's own corruption mechanism was verified the same way: with the magic-number patch made a no-op, the INSERT and VACUUM failure assertions correctly report failure, confirming the test actually exercises the failure path rather than passing regardless.