Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/web/content/docs/core/store.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ type MetadataValue = string | number | boolean | string[];
type Metadata = Readonly<Record<string, MetadataValue>>;
```

Metadata values are `string`, `number`, `boolean`, or `string[]`. That is the intersection of what the seven providers accept. Each adapter adds a few rules of its own. Qdrant, Vectorize, and Upstash in metadata mode reject metadata under `_id` or `_namespace` and a namespace that holds `/`. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`.
Metadata values are `string`, `number`, `boolean`, or `string[]`. That is the intersection of what the seven providers accept. Each adapter adds a few rules of its own. Qdrant and Upstash in metadata mode reject metadata under `_id` or `_namespace`, and Vectorize rejects metadata under `_id`. All three reject a namespace that holds `/`, and Upstash in metadata mode also rejects one that holds a backslash or both quote characters. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`.

## Scores

Expand Down
15 changes: 13 additions & 2 deletions apps/web/content/docs/guides/migration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,20 @@ Moving to Supabase needs a setup step the others do not: install `sql/supabase.s

## Copy the data

The SDK does not move records for you. The SDK has no verb that lists records, so export ids with the provider's own scroll or list call through `store.raw` (Qdrant `scroll`, Pinecone `listPaginated`, Upstash `range`, a keyset `SELECT` on pgvector or Supabase, a paged `FT.SEARCH` on Redis, `listVectors` on Vectorize), then `fetch` them in pages of 1,000 and `upsert` into the new index. The adapter batches upserts, so pass as many records per call as fit in memory.
The SDK does not move records for you. The SDK has no verb that lists records, so export ids with the provider's own scroll or list call through `store.raw` (Qdrant `scroll`, Pinecone `listPaginated`, Upstash `range`, a keyset `SELECT` on pgvector or Supabase, a paged `FT.SEARCH` on Redis, `listVectors` on Vectorize), then `fetch` them in pages of 1,000 with `{ includeVector: true }` and `upsert` into the new index. `fetch` leaves vectors out unless you ask for them, and an `upsert` of a record without one fails (on pgvector and Supabase with `invalid_argument`). The adapter batches upserts, so pass as many records per call as fit in memory.

Record ids, vectors, and metadata round-trip unchanged. Every provider accepts only `string`, `number`, `boolean`, and `string[]` metadata values. Each adapter adds a few rules of its own. Qdrant, Vectorize, and Upstash in metadata mode reject metadata under `_id` or `_namespace` and a namespace that holds `/`. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`.
```ts
const page = await source.fetch(ids, { includeVector: true });
if (!page.ok) {
throw new Error(page.error.message);
}
const copied = await target.upsert(page.value);
if (!copied.ok) {
throw new Error(copied.error.message);
}
```

Record ids and metadata round-trip unchanged. Vectors can come back with less precision than you wrote, because vector databases generally store 32-bit floats: on pgvector and Supabase `0.123456789` returns as `0.12345679`. A Qdrant collection with the cosine metric also stores each vector normalized to unit length. Every provider accepts only `string`, `number`, `boolean`, and `string[]` metadata values. Each adapter adds a few rules of its own. Qdrant and Upstash in metadata mode reject metadata under `_id` or `_namespace`, and Vectorize rejects metadata under `_id`. All three reject a namespace that holds `/`, and Upstash in metadata mode also rejects one that holds a backslash or both quote characters. Redis rejects a value whose type contradicts the declared field type, and an index name that holds `:`.

## Check what each provider stores

Expand Down
2 changes: 1 addition & 1 deletion apps/web/content/docs/guides/testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ bun run test:live

The suite skips providers without a variable. For each provider it creates a temporary index, runs the store verbs, the record verbs, and the `eq`, `gt`, `isIn`, `notIn`, and `not` builders, and deletes the index. The pgvector and Supabase adapters also run on PGlite in the unit suite. `QDRANT_API_KEY` is optional for a local Qdrant and required for Qdrant Cloud.

Three providers ask for more. Supabase needs the SQL functions installed before the suite runs, and the service role key because the suite creates and drops tables. Upstash cannot create an index at all, so the adapter creates a namespace inside the index your URL and token point at. Point them at a scratch index with dimension 3 and the cosine similarity function, which is what the conformance suite asks for. The Vectorize token needs the Vectorize permission, and that run skips the `delete({ all: true })` case, because Vectorize has no call that empties a namespace. `REDIS_URL` has to point at a server that carries the query engine and JSON, such as Redis 8 or Redis Stack; the Redis run adds cases for the default namespace, `exists`, list fields, and delete by filter.
Three providers ask for more. Supabase needs the SQL functions installed before the suite runs, and the service role key because the suite creates and drops tables. Upstash cannot create an index at all, so a vecstore index is a namespace inside the index your URL and token point at, and Upstash creates that namespace on the first write. Point them at a scratch index with dimension 3 and the cosine similarity function, which is what the conformance suite asks for. The Vectorize token needs the Vectorize permission, and that run skips the `delete({ all: true })` case, because Vectorize has no call that empties a namespace. `REDIS_URL` has to point at a server that carries the query engine and JSON, such as Redis 8 or Redis Stack; the Redis run adds cases for the default namespace, `exists`, list fields, and delete by filter.
4 changes: 4 additions & 0 deletions apps/web/content/docs/providers/differences.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,7 @@ The compilers agree on scalar fields with present values. Two cases differ acros
- Redis tag matching is case-sensitive, so `eq("genre", "Drama")` does not match `drama`.
- Redis returns the vector distance as the score, where lower is closer, for every metric. See [Redis](/docs/providers/redis).
- Redis `listIndexes` returns every index in the database, including indexes another application created.

## Metadata keys that contain a dot

Qdrant, Upstash, and Vectorize read a `.` in a filter field as a step into a nested object, and the adapters store metadata flat. A filter on a key that contains a dot, such as `eq("file.name", "a.pdf")`, therefore never sees the value. `eq` matches nothing on those three, and `ne` matches every record on Qdrant and none on Upstash and Vectorize, which evaluate operators against present fields only. pgvector and Supabase look the key up as written, and Redis rejects the field name with `invalid_argument`. Use metadata keys without dots for anything you filter on.
4 changes: 3 additions & 1 deletion apps/web/content/docs/providers/pgvector.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ CREATE TABLE "docs" (

`CREATE EXTENSION` needs a role that can create extensions. If your application role cannot, create the extension yourself before you call `createIndex`.

pgvector's HNSW index holds at most 2,000 dimensions. `createIndex` with a larger `dimension` returns `{ kind: "provider" }` with the message `column cannot have more than 2000 dimensions for hnsw index` and creates nothing. A 3,072-dimension model such as OpenAI's `text-embedding-3-large` needs its output shortened to 2,000 dimensions or fewer for this adapter.

An index name can be at most 49 bytes, so that the `_embedding_idx` and `_metadata_idx` index names stay under the 63-byte Postgres identifier limit. `createIndex` returns `invalid_argument` for a longer name.

When the database has a role named `anon`, as every Supabase project does, `createIndex` also turns on row level security for the new table and adds no policy. The role that ran `createIndex` owns the table and keeps full access. `anon` and `authenticated` see and change nothing through the Supabase REST API until you write a policy, the same as for a table the Supabase adapter creates. On a database without an `anon` role the table is created without row level security. Tables created by version 0.1.4 or earlier have it off; turn it on with `alter table "docs" enable row level security;`.
Expand All @@ -53,7 +55,7 @@ The store caches the metric per table after the first query and forgets it when

## Filtered and large queries

pgvector applies the namespace and the filter after it scans the HNSW index, and a scan stops at `hnsw.ef_search` candidates, 40 by default. Without an iterative scan, a query with a `topK` above 40, a selective filter, or a namespace that holds part of the table returns fewer than `topK` records. On pgvector 0.8.0 or later the adapter turns on an iterative scan for each query, so the scan continues until `topK` records pass. It sets `hnsw.iterative_scan` to `strict_order` for that one statement and keeps a value you set yourself, such as `relaxed_order` on the database.
pgvector applies the namespace and the filter after it scans the HNSW index, and a scan stops at `hnsw.ef_search` candidates, 40 by default. Without an iterative scan, a query with a `topK` above 40, a selective filter, or a namespace that holds part of the table returns fewer than `topK` records. On pgvector 0.8.0 or later the adapter turns on an iterative scan for each query, so the scan continues until `topK` records pass. It sets `hnsw.iterative_scan` to `strict_order` for the current transaction, which under autocommit is the query's own statement, and keeps a value you set yourself, such as `relaxed_order` on the database. If you pass a client inside an explicit transaction, the setting stays on for the rest of that transaction.

On an older pgvector there is no iterative scan and the adapter sends the plain statement. Upgrade the extension, or raise `hnsw.ef_search` for the role or the database. A filter that matches few records can still return fewer than `topK` records on any version, because pgvector stops an iterative scan after `hnsw.max_scan_tuples` tuples, 20,000 by default. The adapter also sends the plain statement to a table with an IVFFlat index, which the SDK never creates.

Expand Down
2 changes: 2 additions & 0 deletions apps/web/content/docs/providers/qdrant.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,6 @@ Namespaces live in the `_namespace` payload key. `createIndex` adds a tenant key
| Metadata | Payload |
| Metric | Set on the collection |

A collection with the cosine metric stores each vector normalized to unit length, so `fetch` and `query` with `includeVector` return the normalized vector, not the one you wrote. Normalize before you compare vectors, or before you copy them into a dot or euclidean index elsewhere.

If you read a collection with another tool, expect the `_id` and `_namespace` keys in the payload.
2 changes: 2 additions & 0 deletions apps/web/content/docs/providers/supabase.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ This is the layout the pgvector adapter uses. A table either adapter created wor

An index name can be at most 49 bytes, so that the `_embedding_idx` and `_metadata_idx` index names stay under the 63-byte Postgres identifier limit. `createIndex` returns `invalid_argument` for a longer name.

pgvector's HNSW index holds at most 2,000 dimensions, so `createIndex` with a larger `dimension` returns `{ kind: "provider" }` and creates nothing, the same as on [pgvector](/docs/providers/pgvector#table-layout).

## Metric detection

`vecstore_query` reads the metric from the HNSW or IVFFlat index definition in `pg_indexes` and defaults to cosine when no index exists. The lookup happens inside the same function call as the search, so a query costs one request.
Expand Down
2 changes: 1 addition & 1 deletion apps/web/content/docs/providers/upstash.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Because the whole name goes into the request path, an index name cannot contain

## Every filtered query has a budget

Upstash gives each query a filtering budget: a number of candidate vectors it can compare against the filter before it falls back to post-filtering. A query whose filter is highly selective can return fewer than `topK` records, so ask for more than you need, or page with a second query.
Upstash gives each query a filtering budget: a number of candidate vectors it can compare against the filter before it falls back to post-filtering. A query whose filter is highly selective can return fewer than `topK` records, so ask for a larger `topK` than you need, or make the filter less selective. `query` has no offset or cursor, so a second identical query returns the same records.

In metadata mode the adapter adds the `_namespace` clause to every query, which makes every query a filtered query. The default namespace compiles to `(HAS NOT FIELD _namespace OR _namespace = '')` so that records written by another tool still match, and an `OR` of two conditions spends more of the budget than a single equality does. Native mode spends none of it on the namespace.

Expand Down
8 changes: 5 additions & 3 deletions packages/vecstore-sdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ const store = createPgvectorStore({ client: pool, schema: "public" });

The client is anything with `query(text, params)` that resolves to `{ rows }`. A `pg` Pool or Client works as is. For postgres.js, wrap `sql.unsafe` so it resolves to `{ rows }`: `{ query: async (text, params) => ({ rows: await sql.unsafe(text, params) }) }`. The test suite does not run against postgres.js.

`createIndex` runs `CREATE EXTENSION IF NOT EXISTS vector` and creates this table plus an HNSW index for the metric and a GIN index on `metadata`, all in one statement, so a failed index build leaves no table behind:
`createIndex` runs `CREATE EXTENSION IF NOT EXISTS vector` and creates this table plus an HNSW index for the metric and a GIN index on `metadata`, all in one statement, so a failed index build leaves no table behind. pgvector's HNSW index holds at most 2,000 dimensions, so a larger `dimension` fails with `provider` and creates nothing, on this adapter and on Supabase:

```sql
CREATE TABLE "docs" (
Expand All @@ -183,7 +183,7 @@ CREATE TABLE "docs" (
);
```

On a database with an `anon` role, such as a Supabase project, `createIndex` also turns on row level security for the table and adds no policy, so the table is closed to the public key until you write one. Queries read the metric from the HNSW or IVFFlat index definition and default to cosine when no index exists. The store caches the metric per table after the first query and forgets it when `createIndex` or `deleteIndex` on the same store touches that table, or when a query on that table fails. On pgvector 0.8.0 or later each query turns on an iterative HNSW scan for its own statement, so a `topK` above 40, a selective filter, or a namespace that holds part of the table still returns `topK` records. Older versions scan at most `hnsw.ef_search` candidates before the namespace and the filter apply, and can return fewer. `listIndexes` returns tables in the schema that have an `embedding vector` column.
On a database with an `anon` role, such as a Supabase project, `createIndex` also turns on row level security for the table and adds no policy, so the table is closed to the public key until you write one. Queries read the metric from the HNSW or IVFFlat index definition and default to cosine when no index exists. The store caches the metric per table after the first query and forgets it when `createIndex` or `deleteIndex` on the same store touches that table, or when a query on that table fails. On pgvector 0.8.0 or later each query turns on an iterative HNSW scan for its transaction, which under autocommit is its own statement, so a `topK` above 40, a selective filter, or a namespace that holds part of the table still returns `topK` records. Older versions scan at most `hnsw.ef_search` candidates before the namespace and the filter apply, and can return fewer. `listIndexes` returns tables in the schema that have an `embedding vector` column.

### Pinecone

Expand Down Expand Up @@ -226,7 +226,7 @@ The index name already uses the one namespace Upstash gives you, so `namespaceMo

`"native"` gives each index and namespace pair its own Upstash namespace, named `{index}` or `{index}~{namespace}`. Ids and metadata are stored as you wrote them, queries carry no namespace filter, and `delete({ all: true })` resets the namespace in one call. Upstash caps namespaces at 100 on the free plan and 10,000 on the paid plans, so choose this mode when you can name your namespaces up front. Nothing migrates between the two modes.

Upstash gives each query a filtering budget and can return fewer than `topK` records when the filter is selective. In metadata mode the adapter scopes every query by namespace, so this applies to every Upstash query. Ask for more than you need, or page with a second query.
Upstash gives each query a filtering budget and can return fewer than `topK` records when the filter is selective. In metadata mode the adapter scopes every query by namespace, so this applies to every Upstash query. Ask for a larger `topK` than you need; `query` has no offset to page with.

Upstash filters are a string rather than an object, so `compileUpstashFilter` validates what it writes. It rejects a field name that breaks Upstash's identifier rule, a string value holding a backslash or both quote characters, and a non-finite number, and the verb returns an `invalid_argument` error.

Expand Down Expand Up @@ -297,6 +297,8 @@ The compilers are exact for scalar fields. Two cases differ across providers:
- Negations (`ne`, `notIn`, `not`) match records that lack the field on Qdrant, pgvector, Supabase, and Redis. Pinecone, Upstash, and Vectorize apply operators to present fields only.
- `eq` and `isIn` on array-valued fields mean "contains" on Qdrant, pgvector, Supabase, and Redis, where each element of a string list is its own tag. Pinecone supports `$in` on list fields and rejects `$eq`. Upstash compares the array itself. Reach an element with the accessors it documents, such as `eq("tags[0]", "x")`. Vectorize indexes string, number, and boolean properties only, so an array-valued field is not filterable there.

A metadata key that contains a `.` cannot be filtered on Qdrant, Upstash, or Vectorize, which read the dot as a step into a nested object. Keep dots out of keys you filter on.

## Testing

Unit tests run with `bun test` against in-memory fakes:
Expand Down
Loading