feat(sync-rules): wildcard schema support for Sync Streams

[henriquekraemer]

Opened at the PowerSync team's request following a positive review of the approach (support request #40221).

Summary

Adds support for a wildcard schema in a Sync Stream's FROM clause, e.g.:

SELECT * FROM "%".assets
WHERE assets.schema() = auth.parameter('tenant_schema')

One stream definition can then span every matching Postgres schema, with the active schema resolved per client. This enables schema-per-tenant replication (a single database with one schema per tenant, e.g. the Rails ros-apartment gem) from a single replication slot, with no per-tenant sync-rules change or redeploy.

How it works

schema() and table_suffix() calls are qualified by a table in scope (a name or alias, resolved through the same scope lookup as column references, so joins and aliases behave as expected: SELECT * FROM "users%" AS users WHERE users.table_suffix() = ...). They compile to a new RowMetadata expression input (sharing a superclass with ColumnInRow, so dependency analysis is unchanged). Sync plans get a matching RowMetadataSqlValue alternative in TableProcessorData, and the evaluators resolve those inputs against the source table of the row being processed, for both bucket data sources and parameter index lookup creators. Since these are function calls rather than column references, they cannot shadow real columns and need no special handling in schema validation.

Changes

• service-sync-rules
  • RowMetadata input in the compiler, parsed from the call case in PostgresToSqlite, with diagnostics for unknown qualified functions, unexpected arguments, unresolved table qualifiers, and table_suffix() on non-wildcard tables.
  • RowMetadataSqlValue alternative in TableProcessorData, serialized as-is in sync plans (no version gating for now, as discussed).
  • Evaluators resolve metadata inputs from the source table, in PreparedStreamBucketDataSource and PreparedParameterIndexLookupCreator.
  • TablePattern: schema-wildcard matching (isSchemaWildcard / schemaPrefix).
• service-module-postgres
  • getQualifiedTableNames discovers tables across all matching schemas for a wildcard-schema pattern (excluding internal schemas), using each table's actual schema for the publication check and the source-table descriptor.

Behavior note

TablePattern.matches() is shared with the older sync rules editions. Previously a "%" schema only matched a schema literally named % (i.e. nothing in practice); it now matches any schema. The metadata functions themselves are only available in the edition-3 compiler.

Tests

• sync-rules: compiler diagnostics, per-schema bucketing, static bucket resolution from the request, metadata as selected columns, and parameter lookups with metadata (the compiler test helper also round-trips plans through serialization).
• module-postgres integration (schema_per_tenant.test.ts, real Postgres + MongoDB storage via describeWithStorage): cross-schema table discovery and per-schema bucket routing.

Changeset included (minor for both packages).

[changeset-bot]

🦋 Changeset detected

Latest commit: 26238d1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages

Name	Type
@powersync/service-errors	Minor
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-convex	Patch
@powersync/service-sync-rules	Minor
@powersync/service-module-postgres	Minor
@powersync/lib-services-framework	Patch
@powersync/service-schema	Patch
@powersync/service-image	Patch
@powersync/service-jpgwire	Patch
@powersync/service-core-tests	Patch
@powersync/service-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[CLAassistant]

All committers have signed the CLA.

[simolus3]

The changes to the Sync Stream compiler and plan look great to me.

Since the plan changes are purely additive, I'm not sure if we need to be concerned about backwards compatibility. The only thing that could go wrong is if a user upgrades to a service version with .schema() and .table_suffix() support, deploys Sync Streams and later downgrades again, they'd get confusing errors because the old version doesn't understand the serialized plan.

Maybe we're fine with this, if not we'd have to bump the plan version to 2 which would make older service versions fail early and with a more recognizable error message. I can open a follow-up PR to update that, it would be good to hear thoughts from @rkistner on this too.

I also haven't checked whether this might need changes for other source databases too.

[stevensJourney]

On plan compatibility: I don't have a strong opinion on whether this needs a plan version bump.

On other source databases: I checked this briefly and I think this is currently Postgres-only at the source discovery layer. The shared sync-rules/compiler changes are generic, but MongoDB, MySQL, MSSQL, and Convex still appear to treat tablePattern.schema as an exact database/schema/default schema. I think we should add an explicit guardrail there. We could log a clear error or throw a hard error when tablePattern.isSchemaWildcard is used for those modules; I’d lean toward throwing so the config does not look valid while the module silently discovers nothing.

[henriquekraemer]

On plan compatibility: I don't have a strong opinion on whether this needs a plan version bump.

On other source databases: I checked this briefly and I think this is currently Postgres-only at the source discovery layer. The shared sync-rules/compiler changes are generic, but MongoDB, MySQL, MSSQL, and Convex still appear to treat tablePattern.schema as an exact database/schema/default schema. I think we should add an explicit guardrail there. We could log a clear error or throw a hard error when tablePattern.isSchemaWildcard is used for those modules; I’d lean toward throwing so the config does not look valid while the module silently discovers nothing.

Something like this? Adding the check at each module's table pattern resolution point. For MongoDB, in MongoSnapshotter.resolveQualifiedTableNames:

private async resolveQualifiedTableNames(
  batch: storage.BucketStorageBatch,
  tablePattern: TablePattern
): Promise<storage.SourceTable[]> {
  if (tablePattern.connectionTag != this.connections.connectionTag) {
    return [];
  }

  if (tablePattern.isSchemaWildcard) {
    throw new ServiceError(
      ErrorCode.PSYNC_Sxxxx,
      'Schema wildcards ("%") in table patterns are not supported for MongoDB connections.'
    );
  }

  const schema = tablePattern.schema;
  // ...
}

Same check in getTablesFromPattern for MySQL (common/schema-utils.ts) and MSSQL (utils/schema.ts), and in the Convex adapter's pattern loop. I'd register a proper error code for it instead of the placeholder above.

One question: do you want this surfaced at validation time too (route API adapters), so the error shows up when deploying the sync config rather than only when replication starts? Happy to add it in both places.

[stevensJourney]

Same check in getTablesFromPattern for MySQL (common/schema-utils.ts) and MSSQL (utils/schema.ts), and in the Convex adapter's pattern loop. I'd register a proper error code for it instead of the placeholder above.

This sounds good to me! From looking at the codes.ts file. I think this range might be applicable

  // ## PSYNC_R22xx: SQL supported feature issues

  // OR Maybe even

  // ## PSYNC_R23xx: SQL schema mismatch issues

One question: do you want this surfaced at validation time too (route API adapters), so the error shows up when deploying the sync config rather than only when replication starts? Happy to add it in both places.

This sounds like it would be very nice to have. I'm not sure how much effort this would entail though. I think throwing an error would be fine for the first release.

[henriquekraemer]

Same check in getTablesFromPattern for MySQL (common/schema-utils.ts) and MSSQL (utils/schema.ts), and in the Convex adapter's pattern loop. I'd register a proper error code for it instead of the placeholder above.

This sounds good to me! From looking at the codes.ts file. I think this range might be applicable

  // ## PSYNC_R22xx: SQL supported feature issues

  // OR Maybe even

  // ## PSYNC_R23xx: SQL schema mismatch issues

One question: do you want this surfaced at validation time too (route API adapters), so the error shows up when deploying the sync config rather than only when replication starts? Happy to add it in both places.

This sounds like it would be very nice to have. I'm not sure how much effort this would entail though. I think throwing an error would be fine for the first release.

Done in 6924d32. Registered PSYNC_R2201 in the PSYNC_R22xx range and added the hard error at each module's table pattern resolution point: MongoSnapshotter.resolveQualifiedTableNames (MongoDB), getTablesFromPattern (MySQL and SQL Server) and ConvexStream.resolveQualifiedTableNames (Convex). The check runs after the connectionTag filter where there is one, so patterns belonging to other connections are unaffected. I'll leave the validation-time surfacing for a follow-up as suggested.

[simolus3]

Maybe we're fine with this, if not we'd have to bump the plan version to 2 which would make older service versions fail early and with a more recognizable error message

We've discussed this offline, and our plan is to bump sync plans to v2 if they use the new functions in expressions. I will open a follow-up PR for that, hopefully sometime next week. That probably doesn't include too many changes in this case, but it also helps us establish the process of increasing sync plan versions, as we've not done that before.

We want to keep this PR open until the follow-up PR is in place, but it looks good to me in its current shape.

[henriquekraemer]

Maybe we're fine with this, if not we'd have to bump the plan version to 2 which would make older service versions fail early and with a more recognizable error message

We've discussed this offline, and our plan is to bump sync plans to v2 if they use the new functions in expressions. I will open a follow-up PR for that, hopefully sometime next week. That probably doesn't include too many changes in this case, but it also helps us establish the process of increasing sync plan versions, as we've not done that before.

We want to keep this PR open until the follow-up PR is in place, but it looks good to me in its current shape.

Sounds great. Happy to rebase once the follow-up lands.

[simolus3]

I'm happy to merge this (and #717 for sync plan versioning before we release this) in its current state.

One thing I wonder is whether it could make sense to also expose the full table name, perhaps via .table_name(), even when no suffix is used? Obviously one can reconstruct it via SELECT * FROM "users_%" AS users WHERE 'users_' || users.table_suffix() == ..., but it still feels like a convenient thing to have in case the auth parameter contains the full table name and it should be relatively easy to add now before we commit to the new sync plan version. What do you think?

[henriquekraemer]

I'm happy to merge this (and #717 for sync plan versioning before we release this) in its current state.

One thing I wonder is whether it could make sense to also expose the full table name, perhaps via .table_name(), even when no suffix is used? Obviously one can reconstruct it via SELECT * FROM "users_%" AS users WHERE 'users_' || users.table_suffix() == ..., but it still feels like a convenient thing to have in case the auth parameter contains the full table name and it should be relatively easy to add now before we commit to the new sync plan version. What do you think?

Good idea, and agreed it is easier to add now. .table_name() returning the full replicated table name (table.name) is a small addition alongside the existing two, so I'll add it. Want me to push it here on #703 so it lands before the v2 plan version in #717?

[simolus3]

Yes, let's add that here. I can then merge this PR and change the target branch for the follow-up to main.

[henriquekraemer]

Yes, let's add that here. I can then merge this PR and change the target branch for the follow-up to main.

Added in 1736a0c: users.table_name() returns the full replicated table name, on any table (no wildcard required, so no warning like table_suffix() has on non-wildcard tables). Changeset updated as well.