Term searches fail with "ResourceOwnerForget called for relcache reference after release started" under delete/rewrite churn; session then cannot REINDEX/DROP the index

[honoka-lulu]

Summary

On a delete/rewrite-heavy corpus, searches for specific terms start failing deterministically with:

ERROR: ResourceOwnerForget called for relcache reference after release started

Once a session has hit the error, that backend is poisoned: a later REINDEX INDEX CONCURRENTLY issued from the same session reaches the final drop phase and fails with cannot DROP INDEX ... because it is being used by active queries in this session (a fresh session succeeds). REINDEX INDEX CONCURRENTLY on the affected index fully restores search — until churn accumulates again.

Environment

• vchord_bm25 0.3.0 (pgrx 0.16.1)
• PostgreSQL 18.3 (Ubuntu pgdg), managed hosting
• pg_tokenizer 0.1.1, vector 0.8.2, vchord 1.1.1 in the same database
• 128 hash-partitioned BM25 indexes (one per partition of a chunk table)

Workload profile that reproduces it

Sync pipelines that periodically re-ingest documents as delete + insert (hundreds to thousands of documents per day per index). Under that churn a given index degrades in days to ~2 weeks: one or more terms become "poisoned" and every search containing such a term fails with the error above, deterministically, while other terms keep working. Across our fleet the auto-remediation log shows 38 REINDEX cycles in 5 weeks (~1/day), and each REINDEX also shrinks the index by 40–60% (e.g. 2569 MB → 1054 MB), so degradation correlates with accumulated delete/append structures rather than corpus size.

Source-level analysis (0.3.0)

We believe there are two stacked problems.

1. The relcache error and the session poisoning come from how the scan state owns its PgRelation (src/index/scan.rs):

• ambeginscan leaks the Scanner into the current memory context with leak_and_drop_on_delete, so the Rust Drop runs from a memory-context destruction callback (scan.rs:64-69).
• amrescan stores query_index: PgRelation::with_lock(index_oid, AccessShareLock) inside that scanner (scan.rs:96-101) — the scan state now owns a relcache reference.
• On the happy path amendscan overwrites the scanner with Scanner::Initial, dropping the PgRelation at a legal time (scan.rs:147-151).
• On any error inside the scan, amendscan never runs. The PgRelation is then closed from the memory-context callback during transaction abort, i.e. while the resource owner's release is already in progress → ResourceOwnerForget called for relcache reference after release started, and the reference is effectively leaked in that backend. That leaked reference is exactly what later makes REINDEX CONCURRENTLY's drop phase in the same session report the index as "being used by active queries in this session".

A possible fix: amrescan already receives the index oid inside the bm25query datum, but the executor has the same index open as scan->indexRelation. Validating the datum oid against RelationGetRelid(scan->indexRelation) and reusing that relation would remove the extra relcache reference entirely (and error cleanly on a mismatch). Alternatively, the relation must be closed at a point that is guaranteed to run before resource-owner release, not from a context callback.

2. The underlying per-term failure looks like a data-dependent panic in the posting read path under churned segments. We have not isolated a minimal corpus yet, but the determinism per term, the fix-by-REINDEX, and the correlation with delete+append volume all point at an edge state in the sealed/append posting structures (src/segment/posting/reader.rs has several unwrap()s and index arithmetic such as unfulled_docid[unfulled_doc_cnt - 1] in append.rs that panic on empty/edge states — the same class as the fixed #80). Because of problem 1, whatever the first error is, the client-visible message becomes the relcache error and the backend is left poisoned — so fixing problem 1 should also surface the true underlying error and make this diagnosable.

Repro sketch

We cannot share the corpus, but the shape is:

1. Build a BM25 index; ingest N documents.
2. Loop for days: delete a large fraction, re-insert variants (delete + insert, not update), let inserts go through the growing segment/append path.
3. Periodically search a fixed set of common terms; eventually one term starts erroring with the message above while others still work.
4. In the erroring session, try REINDEX INDEX CONCURRENTLY — the drop phase fails "in use by active queries in this session"; from a fresh session it succeeds and search recovers.

Happy to run diagnostics against our production corpus (page inspector output for a poisoned term's postings, etc.) if that helps.