From d20f01bfa0f63192a1e55098b16d1e8aff2b5b7b Mon Sep 17 00:00:00 2001 From: PunGrumpy <108584943+PunGrumpy@users.noreply.github.com> Date: Sun, 18 Oct 2026 16:30:08 +0700 Subject: [PATCH 1/5] docs: fetch vectors when copying records between providers The copy recipe fetched records and upserted them, but fetch leaves vectors out unless includeVector is set. Each fetched record then had an empty vector, and upsert rejected it with invalid_argument. The recipe now passes includeVector, shows the call, and no longer says vectors round-trip unchanged. Vector databases generally store 32-bit floats, and a Qdrant collection with the cosine metric stores unit-length vectors, so the Qdrant page says so too. --- apps/web/content/docs/guides/migration.mdx | 15 +++++++++++++-- apps/web/content/docs/providers/qdrant.mdx | 2 ++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/apps/web/content/docs/guides/migration.mdx b/apps/web/content/docs/guides/migration.mdx index 380d4bd..30c4923 100644 --- a/apps/web/content/docs/guides/migration.mdx +++ b/apps/web/content/docs/guides/migration.mdx @@ -71,9 +71,20 @@ Moving to Supabase needs a setup step the others do not: install `sql/supabase.s ## Copy the data -The SDK does not move records for you. The SDK has no verb that lists records, so export ids with the provider's own scroll or list call through `store.raw` (Qdrant `scroll`, Pinecone `listPaginated`, Upstash `range`, a keyset `SELECT` on pgvector or Supabase, a paged `FT.SEARCH` on Redis, `listVectors` on Vectorize), then `fetch` them in pages of 1,000 and `upsert` into the new index. The adapter batches upserts, so pass as many records per call as fit in memory. +The SDK does not move records for you. The SDK has no verb that lists records, so export ids with the provider's own scroll or list call through `store.raw` (Qdrant `scroll`, Pinecone `listPaginated`, Upstash `range`, a keyset `SELECT` on pgvector or Supabase, a paged `FT.SEARCH` on Redis, `listVectors` on Vectorize), then `fetch` them in pages of 1,000 with `{ includeVector: true }` and `upsert` into the new index. `fetch` leaves vectors out unless you ask for them, and an `upsert` of a record without one fails (on pgvector and Supabase with `invalid_argument`). The adapter batches upserts, so pass as many records per call as fit in memory. -Record ids, vectors, and metadata round-trip unchanged. Every provider accepts only `string`, `number`, `boolean`, and `string[]` metadata values. Each adapter adds a few rules of its own. Qdrant, Vectorize, and Upstash in metadata mode reject metadata under `_id` or `_namespace` and a namespace that holds `/`. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`. +```ts +const page = await source.fetch(ids, { includeVector: true }); +if (!page.ok) { + throw new Error(page.error.message); +} +const copied = await target.upsert(page.value); +if (!copied.ok) { + throw new Error(copied.error.message); +} +``` + +Record ids and metadata round-trip unchanged. Vectors can come back with less precision than you wrote, because vector databases generally store 32-bit floats: on pgvector and Supabase `0.123456789` returns as `0.12345679`. A Qdrant collection with the cosine metric also stores each vector normalized to unit length. Every provider accepts only `string`, `number`, `boolean`, and `string[]` metadata values. Each adapter adds a few rules of its own. Qdrant, Vectorize, and Upstash in metadata mode reject metadata under `_id` or `_namespace` and a namespace that holds `/`. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`. ## Check what each provider stores diff --git a/apps/web/content/docs/providers/qdrant.mdx b/apps/web/content/docs/providers/qdrant.mdx index 310e061..1286fdf 100644 --- a/apps/web/content/docs/providers/qdrant.mdx +++ b/apps/web/content/docs/providers/qdrant.mdx @@ -40,4 +40,6 @@ Namespaces live in the `_namespace` payload key. `createIndex` adds a tenant key | Metadata | Payload | | Metric | Set on the collection | +A collection with the cosine metric stores each vector normalized to unit length, so `fetch` and `query` with `includeVector` return the normalized vector, not the one you wrote. Normalize before you compare vectors, or before you copy them into a dot or euclidean index elsewhere. + If you read a collection with another tool, expect the `_id` and `_namespace` keys in the payload. From 02f0861475aad0fabc9456c7e7b70366facd2859 Mon Sep 17 00:00:00 2001 From: PunGrumpy <108584943+PunGrumpy@users.noreply.github.com> Date: Sun, 18 Oct 2026 16:30:36 +0700 Subject: [PATCH 2/5] docs: document pgvector's 2,000-dimension hnsw limit pgvector's HNSW index holds at most 2,000 dimensions. createIndex with a larger dimension returns a provider error and creates nothing, on both the pgvector and Supabase adapters. That rules out 3,072-dimension models such as text-embedding-3-large, and the error kind is provider, not the invalid_argument a reader would expect for a dimension problem. The pgvector and Supabase pages and the README now say so. --- apps/web/content/docs/providers/pgvector.mdx | 2 ++ apps/web/content/docs/providers/supabase.mdx | 2 ++ packages/vecstore-sdk/README.md | 2 +- 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/apps/web/content/docs/providers/pgvector.mdx b/apps/web/content/docs/providers/pgvector.mdx index f0ea473..bdc0212 100644 --- a/apps/web/content/docs/providers/pgvector.mdx +++ b/apps/web/content/docs/providers/pgvector.mdx @@ -39,6 +39,8 @@ CREATE TABLE "docs" ( `CREATE EXTENSION` needs a role that can create extensions. If your application role cannot, create the extension yourself before you call `createIndex`. +pgvector's HNSW index holds at most 2,000 dimensions. `createIndex` with a larger `dimension` returns `{ kind: "provider" }` with the message `column cannot have more than 2000 dimensions for hnsw index` and creates nothing. A 3,072-dimension model such as OpenAI's `text-embedding-3-large` needs its output shortened to 2,000 dimensions or fewer for this adapter. + An index name can be at most 49 bytes, so that the `_embedding_idx` and `_metadata_idx` index names stay under the 63-byte Postgres identifier limit. `createIndex` returns `invalid_argument` for a longer name. When the database has a role named `anon`, as every Supabase project does, `createIndex` also turns on row level security for the new table and adds no policy. The role that ran `createIndex` owns the table and keeps full access. `anon` and `authenticated` see and change nothing through the Supabase REST API until you write a policy, the same as for a table the Supabase adapter creates. On a database without an `anon` role the table is created without row level security. Tables created by version 0.1.4 or earlier have it off; turn it on with `alter table "docs" enable row level security;`. diff --git a/apps/web/content/docs/providers/supabase.mdx b/apps/web/content/docs/providers/supabase.mdx index d6139ac..6f5772f 100644 --- a/apps/web/content/docs/providers/supabase.mdx +++ b/apps/web/content/docs/providers/supabase.mdx @@ -71,6 +71,8 @@ This is the layout the pgvector adapter uses. A table either adapter created wor An index name can be at most 49 bytes, so that the `_embedding_idx` and `_metadata_idx` index names stay under the 63-byte Postgres identifier limit. `createIndex` returns `invalid_argument` for a longer name. +pgvector's HNSW index holds at most 2,000 dimensions, so `createIndex` with a larger `dimension` returns `{ kind: "provider" }` and creates nothing, the same as on [pgvector](/docs/providers/pgvector#table-layout). + ## Metric detection `vecstore_query` reads the metric from the HNSW or IVFFlat index definition in `pg_indexes` and defaults to cosine when no index exists. The lookup happens inside the same function call as the search, so a query costs one request. diff --git a/packages/vecstore-sdk/README.md b/packages/vecstore-sdk/README.md index 88d8e60..1ab0af8 100644 --- a/packages/vecstore-sdk/README.md +++ b/packages/vecstore-sdk/README.md @@ -171,7 +171,7 @@ const store = createPgvectorStore({ client: pool, schema: "public" }); The client is anything with `query(text, params)` that resolves to `{ rows }`. A `pg` Pool or Client works as is. For postgres.js, wrap `sql.unsafe` so it resolves to `{ rows }`: `{ query: async (text, params) => ({ rows: await sql.unsafe(text, params) }) }`. The test suite does not run against postgres.js. -`createIndex` runs `CREATE EXTENSION IF NOT EXISTS vector` and creates this table plus an HNSW index for the metric and a GIN index on `metadata`, all in one statement, so a failed index build leaves no table behind: +`createIndex` runs `CREATE EXTENSION IF NOT EXISTS vector` and creates this table plus an HNSW index for the metric and a GIN index on `metadata`, all in one statement, so a failed index build leaves no table behind. pgvector's HNSW index holds at most 2,000 dimensions, so a larger `dimension` fails with `provider` and creates nothing, on this adapter and on Supabase: ```sql CREATE TABLE "docs" ( From f5efc00d06c57df5ad7f7466e4edcc6510d54d0e Mon Sep 17 00:00:00 2001 From: PunGrumpy <108584943+PunGrumpy@users.noreply.github.com> Date: Sun, 18 Oct 2026 16:31:09 +0700 Subject: [PATCH 3/5] docs: correct the reserved keys, upstash namespaces, and paging advice Vectorize reserves only the _id metadata key, not _namespace, and the pages grouped it with Qdrant and Upstash. Upstash in metadata mode also rejects a namespace holding a backslash or both quote characters, so the sentence now says that. The testing guide said the Upstash adapter creates a namespace, but Upstash creates it on the first write. The Upstash page and the README told readers to page with a second query, and query has no offset or cursor, so a second query returns the same records. Both now say to ask for a larger topK or loosen the filter. --- apps/web/content/docs/core/store.mdx | 2 +- apps/web/content/docs/guides/migration.mdx | 2 +- apps/web/content/docs/guides/testing.mdx | 2 +- apps/web/content/docs/providers/upstash.mdx | 2 +- packages/vecstore-sdk/README.md | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/apps/web/content/docs/core/store.mdx b/apps/web/content/docs/core/store.mdx index 568b044..ca9afe0 100644 --- a/apps/web/content/docs/core/store.mdx +++ b/apps/web/content/docs/core/store.mdx @@ -42,7 +42,7 @@ type MetadataValue = string | number | boolean | string[]; type Metadata = Readonly>; ``` -Metadata values are `string`, `number`, `boolean`, or `string[]`. That is the intersection of what the seven providers accept. Each adapter adds a few rules of its own. Qdrant, Vectorize, and Upstash in metadata mode reject metadata under `_id` or `_namespace` and a namespace that holds `/`. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`. +Metadata values are `string`, `number`, `boolean`, or `string[]`. That is the intersection of what the seven providers accept. Each adapter adds a few rules of its own. Qdrant and Upstash in metadata mode reject metadata under `_id` or `_namespace`, and Vectorize rejects metadata under `_id`. All three reject a namespace that holds `/`, and Upstash in metadata mode also rejects one that holds a backslash or both quote characters. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`. ## Scores diff --git a/apps/web/content/docs/guides/migration.mdx b/apps/web/content/docs/guides/migration.mdx index 30c4923..cf8725a 100644 --- a/apps/web/content/docs/guides/migration.mdx +++ b/apps/web/content/docs/guides/migration.mdx @@ -84,7 +84,7 @@ if (!copied.ok) { } ``` -Record ids and metadata round-trip unchanged. Vectors can come back with less precision than you wrote, because vector databases generally store 32-bit floats: on pgvector and Supabase `0.123456789` returns as `0.12345679`. A Qdrant collection with the cosine metric also stores each vector normalized to unit length. Every provider accepts only `string`, `number`, `boolean`, and `string[]` metadata values. Each adapter adds a few rules of its own. Qdrant, Vectorize, and Upstash in metadata mode reject metadata under `_id` or `_namespace` and a namespace that holds `/`. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`. +Record ids and metadata round-trip unchanged. Vectors can come back with less precision than you wrote, because vector databases generally store 32-bit floats: on pgvector and Supabase `0.123456789` returns as `0.12345679`. A Qdrant collection with the cosine metric also stores each vector normalized to unit length. Every provider accepts only `string`, `number`, `boolean`, and `string[]` metadata values. Each adapter adds a few rules of its own. Qdrant and Upstash in metadata mode reject metadata under `_id` or `_namespace`, and Vectorize rejects metadata under `_id`. All three reject a namespace that holds `/`, and Upstash in metadata mode also rejects one that holds a backslash or both quote characters. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`. ## Check what each provider stores diff --git a/apps/web/content/docs/guides/testing.mdx b/apps/web/content/docs/guides/testing.mdx index 60586a5..5a18e6d 100644 --- a/apps/web/content/docs/guides/testing.mdx +++ b/apps/web/content/docs/guides/testing.mdx @@ -36,4 +36,4 @@ bun run test:live The suite skips providers without a variable. For each provider it creates a temporary index, runs the store verbs, the record verbs, and the `eq`, `gt`, `isIn`, `notIn`, and `not` builders, and deletes the index. The pgvector and Supabase adapters also run on PGlite in the unit suite. `QDRANT_API_KEY` is optional for a local Qdrant and required for Qdrant Cloud. -Three providers ask for more. Supabase needs the SQL functions installed before the suite runs, and the service role key because the suite creates and drops tables. Upstash cannot create an index at all, so the adapter creates a namespace inside the index your URL and token point at. Point them at a scratch index with dimension 3 and the cosine similarity function, which is what the conformance suite asks for. The Vectorize token needs the Vectorize permission, and that run skips the `delete({ all: true })` case, because Vectorize has no call that empties a namespace. `REDIS_URL` has to point at a server that carries the query engine and JSON, such as Redis 8 or Redis Stack; the Redis run adds cases for the default namespace, `exists`, list fields, and delete by filter. +Three providers ask for more. Supabase needs the SQL functions installed before the suite runs, and the service role key because the suite creates and drops tables. Upstash cannot create an index at all, so a vecstore index is a namespace inside the index your URL and token point at, and Upstash creates that namespace on the first write. Point them at a scratch index with dimension 3 and the cosine similarity function, which is what the conformance suite asks for. The Vectorize token needs the Vectorize permission, and that run skips the `delete({ all: true })` case, because Vectorize has no call that empties a namespace. `REDIS_URL` has to point at a server that carries the query engine and JSON, such as Redis 8 or Redis Stack; the Redis run adds cases for the default namespace, `exists`, list fields, and delete by filter. diff --git a/apps/web/content/docs/providers/upstash.mdx b/apps/web/content/docs/providers/upstash.mdx index 1d116bc..b85a873 100644 --- a/apps/web/content/docs/providers/upstash.mdx +++ b/apps/web/content/docs/providers/upstash.mdx @@ -78,7 +78,7 @@ Because the whole name goes into the request path, an index name cannot contain ## Every filtered query has a budget -Upstash gives each query a filtering budget: a number of candidate vectors it can compare against the filter before it falls back to post-filtering. A query whose filter is highly selective can return fewer than `topK` records, so ask for more than you need, or page with a second query. +Upstash gives each query a filtering budget: a number of candidate vectors it can compare against the filter before it falls back to post-filtering. A query whose filter is highly selective can return fewer than `topK` records, so ask for a larger `topK` than you need, or make the filter less selective. `query` has no offset or cursor, so a second identical query returns the same records. In metadata mode the adapter adds the `_namespace` clause to every query, which makes every query a filtered query. The default namespace compiles to `(HAS NOT FIELD _namespace OR _namespace = '')` so that records written by another tool still match, and an `OR` of two conditions spends more of the budget than a single equality does. Native mode spends none of it on the namespace. diff --git a/packages/vecstore-sdk/README.md b/packages/vecstore-sdk/README.md index 1ab0af8..d9a4164 100644 --- a/packages/vecstore-sdk/README.md +++ b/packages/vecstore-sdk/README.md @@ -226,7 +226,7 @@ The index name already uses the one namespace Upstash gives you, so `namespaceMo `"native"` gives each index and namespace pair its own Upstash namespace, named `{index}` or `{index}~{namespace}`. Ids and metadata are stored as you wrote them, queries carry no namespace filter, and `delete({ all: true })` resets the namespace in one call. Upstash caps namespaces at 100 on the free plan and 10,000 on the paid plans, so choose this mode when you can name your namespaces up front. Nothing migrates between the two modes. -Upstash gives each query a filtering budget and can return fewer than `topK` records when the filter is selective. In metadata mode the adapter scopes every query by namespace, so this applies to every Upstash query. Ask for more than you need, or page with a second query. +Upstash gives each query a filtering budget and can return fewer than `topK` records when the filter is selective. In metadata mode the adapter scopes every query by namespace, so this applies to every Upstash query. Ask for a larger `topK` than you need; `query` has no offset to page with. Upstash filters are a string rather than an object, so `compileUpstashFilter` validates what it writes. It rejects a field name that breaks Upstash's identifier rule, a string value holding a backslash or both quote characters, and a non-finite number, and the verb returns an `invalid_argument` error. From f8b6a05f5335f52265e2f9d04fd8bb7ab5c1aec6 Mon Sep 17 00:00:00 2001 From: PunGrumpy <108584943+PunGrumpy@users.noreply.github.com> Date: Sun, 18 Oct 2026 16:31:39 +0700 Subject: [PATCH 4/5] docs: say how long the iterative scan setting lasts The adapter sets hnsw.iterative_scan with set_config(..., true), which scopes the setting to the current transaction. Under autocommit that is the query's own statement, but a caller who passes a client inside an explicit transaction keeps strict_order on until the transaction ends. The pgvector page and the README said the setting lasted one statement. --- apps/web/content/docs/providers/pgvector.mdx | 2 +- packages/vecstore-sdk/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/web/content/docs/providers/pgvector.mdx b/apps/web/content/docs/providers/pgvector.mdx index bdc0212..8201468 100644 --- a/apps/web/content/docs/providers/pgvector.mdx +++ b/apps/web/content/docs/providers/pgvector.mdx @@ -55,7 +55,7 @@ The store caches the metric per table after the first query and forgets it when ## Filtered and large queries -pgvector applies the namespace and the filter after it scans the HNSW index, and a scan stops at `hnsw.ef_search` candidates, 40 by default. Without an iterative scan, a query with a `topK` above 40, a selective filter, or a namespace that holds part of the table returns fewer than `topK` records. On pgvector 0.8.0 or later the adapter turns on an iterative scan for each query, so the scan continues until `topK` records pass. It sets `hnsw.iterative_scan` to `strict_order` for that one statement and keeps a value you set yourself, such as `relaxed_order` on the database. +pgvector applies the namespace and the filter after it scans the HNSW index, and a scan stops at `hnsw.ef_search` candidates, 40 by default. Without an iterative scan, a query with a `topK` above 40, a selective filter, or a namespace that holds part of the table returns fewer than `topK` records. On pgvector 0.8.0 or later the adapter turns on an iterative scan for each query, so the scan continues until `topK` records pass. It sets `hnsw.iterative_scan` to `strict_order` for the current transaction, which under autocommit is the query's own statement, and keeps a value you set yourself, such as `relaxed_order` on the database. If you pass a client inside an explicit transaction, the setting stays on for the rest of that transaction. On an older pgvector there is no iterative scan and the adapter sends the plain statement. Upgrade the extension, or raise `hnsw.ef_search` for the role or the database. A filter that matches few records can still return fewer than `topK` records on any version, because pgvector stops an iterative scan after `hnsw.max_scan_tuples` tuples, 20,000 by default. The adapter also sends the plain statement to a table with an IVFFlat index, which the SDK never creates. diff --git a/packages/vecstore-sdk/README.md b/packages/vecstore-sdk/README.md index d9a4164..e0bc5e0 100644 --- a/packages/vecstore-sdk/README.md +++ b/packages/vecstore-sdk/README.md @@ -183,7 +183,7 @@ CREATE TABLE "docs" ( ); ``` -On a database with an `anon` role, such as a Supabase project, `createIndex` also turns on row level security for the table and adds no policy, so the table is closed to the public key until you write one. Queries read the metric from the HNSW or IVFFlat index definition and default to cosine when no index exists. The store caches the metric per table after the first query and forgets it when `createIndex` or `deleteIndex` on the same store touches that table, or when a query on that table fails. On pgvector 0.8.0 or later each query turns on an iterative HNSW scan for its own statement, so a `topK` above 40, a selective filter, or a namespace that holds part of the table still returns `topK` records. Older versions scan at most `hnsw.ef_search` candidates before the namespace and the filter apply, and can return fewer. `listIndexes` returns tables in the schema that have an `embedding vector` column. +On a database with an `anon` role, such as a Supabase project, `createIndex` also turns on row level security for the table and adds no policy, so the table is closed to the public key until you write one. Queries read the metric from the HNSW or IVFFlat index definition and default to cosine when no index exists. The store caches the metric per table after the first query and forgets it when `createIndex` or `deleteIndex` on the same store touches that table, or when a query on that table fails. On pgvector 0.8.0 or later each query turns on an iterative HNSW scan for its transaction, which under autocommit is its own statement, so a `topK` above 40, a selective filter, or a namespace that holds part of the table still returns `topK` records. Older versions scan at most `hnsw.ef_search` candidates before the namespace and the filter apply, and can return fewer. `listIndexes` returns tables in the schema that have an `embedding vector` column. ### Pinecone From 1c74f03d01abbecd7855e8a9373f64d4695924af Mon Sep 17 00:00:00 2001 From: PunGrumpy <108584943+PunGrumpy@users.noreply.github.com> Date: Sun, 18 Oct 2026 16:32:12 +0700 Subject: [PATCH 5/5] docs: say that a dot in a metadata key means nesting on three providers Qdrant, Upstash, and Vectorize read a dot in a filter field name as a step into a nested object, and the adapters store metadata flat. A filter on a key such as file.name never sees the value there. The differences page and the README now say to keep dots out of keys you filter on. This rests on the providers' filter documentation and the compilers, not on a live run. --- apps/web/content/docs/providers/differences.mdx | 4 ++++ packages/vecstore-sdk/README.md | 2 ++ 2 files changed, 6 insertions(+) diff --git a/apps/web/content/docs/providers/differences.mdx b/apps/web/content/docs/providers/differences.mdx index 4faecc7..2e6f995 100644 --- a/apps/web/content/docs/providers/differences.mdx +++ b/apps/web/content/docs/providers/differences.mdx @@ -33,3 +33,7 @@ The compilers agree on scalar fields with present values. Two cases differ acros - Redis tag matching is case-sensitive, so `eq("genre", "Drama")` does not match `drama`. - Redis returns the vector distance as the score, where lower is closer, for every metric. See [Redis](/docs/providers/redis). - Redis `listIndexes` returns every index in the database, including indexes another application created. + +## Metadata keys that contain a dot + +Qdrant, Upstash, and Vectorize read a `.` in a filter field as a step into a nested object, and the adapters store metadata flat. A filter on a key that contains a dot, such as `eq("file.name", "a.pdf")`, therefore never sees the value. `eq` matches nothing on those three, and `ne` matches every record on Qdrant and none on Upstash and Vectorize, which evaluate operators against present fields only. pgvector and Supabase look the key up as written, and Redis rejects the field name with `invalid_argument`. Use metadata keys without dots for anything you filter on. diff --git a/packages/vecstore-sdk/README.md b/packages/vecstore-sdk/README.md index e0bc5e0..ca952c3 100644 --- a/packages/vecstore-sdk/README.md +++ b/packages/vecstore-sdk/README.md @@ -297,6 +297,8 @@ The compilers are exact for scalar fields. Two cases differ across providers: - Negations (`ne`, `notIn`, `not`) match records that lack the field on Qdrant, pgvector, Supabase, and Redis. Pinecone, Upstash, and Vectorize apply operators to present fields only. - `eq` and `isIn` on array-valued fields mean "contains" on Qdrant, pgvector, Supabase, and Redis, where each element of a string list is its own tag. Pinecone supports `$in` on list fields and rejects `$eq`. Upstash compares the array itself. Reach an element with the accessors it documents, such as `eq("tags[0]", "x")`. Vectorize indexes string, number, and boolean properties only, so an array-valued field is not filterable there. +A metadata key that contains a `.` cannot be filtered on Qdrant, Upstash, or Vectorize, which read the dot as a step into a nested object. Keep dots out of keys you filter on. + ## Testing Unit tests run with `bun test` against in-memory fakes: