Repository navigation
docs: fix the copy recipe that drops vectors, and document the limits the code already has - #59
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 25 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
commit: |
The copy recipe fetched records and upserted them, but fetch leaves vectors out unless includeVector is set. Each fetched record then had an empty vector, and upsert rejected it with invalid_argument. The recipe now passes includeVector, shows the call, and no longer says vectors round-trip unchanged. Vector databases generally store 32-bit floats, and a Qdrant collection with the cosine metric stores unit-length vectors, so the Qdrant page says so too.
pgvector's HNSW index holds at most 2,000 dimensions. createIndex with a larger dimension returns a provider error and creates nothing, on both the pgvector and Supabase adapters. That rules out 3,072-dimension models such as text-embedding-3-large, and the error kind is provider, not the invalid_argument a reader would expect for a dimension problem. The pgvector and Supabase pages and the README now say so.
Vectorize reserves only the _id metadata key, not _namespace, and the pages grouped it with Qdrant and Upstash. Upstash in metadata mode also rejects a namespace holding a backslash or both quote characters, so the sentence now says that. The testing guide said the Upstash adapter creates a namespace, but Upstash creates it on the first write. The Upstash page and the README told readers to page with a second query, and query has no offset or cursor, so a second query returns the same records. Both now say to ask for a larger topK or loosen the filter.
The adapter sets hnsw.iterative_scan with set_config(..., true), which scopes the setting to the current transaction. Under autocommit that is the query's own statement, but a caller who passes a client inside an explicit transaction keeps strict_order on until the transaction ends. The pgvector page and the README said the setting lasted one statement.
Qdrant, Upstash, and Vectorize read a dot in a filter field name as a step into a nested object, and the adapters store metadata flat. A filter on a key such as file.name never sees the value there. The differences page and the README now say to keep dots out of keys you filter on. This rests on the providers' filter documentation and the compilers, not on a live run.
PunGrumpy
force-pushed
the
docs/copy-recipe-and-limits
branch
from
October 18, 2026 09:48
bde46cf to
1c74f03
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #58 (and the stack under it: #57, #55, #54). Review the last five commits, from
docs: fetch vectors when copying records between providers.Background
The migration guide's "Copy the data" recipe didn't work. It said to
fetchthe records andupsertthem into the new index, and that ids, vectors and metadata round-trip unchanged.fetchleaves vectors out unless you pass{ includeVector: true }, so every copied record hadvector: []; on PGlite theupsertreturnedinvalid_argument "vector must have at least 1 dimension". Vectors also don't round-trip exactly: pgvector stores 32-bit floats, and Qdrant normalizes vectors in a cosine collection.A few other statements were missing or contradicted the code:
createIndexwithdimension: 2001returnsproviderand creates nothing, on pgvector and Supabase, which rules out 3,072-dimension models liketext-embedding-3-large._namespace; it only reserves_id. They now also list the Upstash namespace rule from fix(upstash): keep namespaces apart on fetch and reject names the adapter cannot use #54.queryhas no offset or cursor.set_config(…, true)scopes it to the transaction..in a filter field as nesting, so a flat key likefile.namecan't be filtered there.Summary
includeVector: truein the recipe, a short copy example, and an accurate precision note. The Qdrant page says cosine collections return normalized vectors.Docs only; no code change.
Verification
bun run checkandbun run typecheckare clean, and an uncachedturbo run build --filter=webcompiles and generates all 64 pages.includeVector, the 2,000-dimension error, and the transaction scope of the setting were observed on PGlite before writing the sentences.Checklist
vecstore-sdk(docs only; the site deploys with the next publish)