Skip to content

Add Support for Vector Data Type in SQL for REST#3677

Open
RubenCerna2079 wants to merge 37 commits into
mainfrom
dev/rubencerna/first-changes-vector-type
Open

Add Support for Vector Data Type in SQL for REST#3677
RubenCerna2079 wants to merge 37 commits into
mainfrom
dev/rubencerna/first-changes-vector-type

Conversation

@RubenCerna2079

@RubenCerna2079 RubenCerna2079 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Why make this change?

What is this change?

These changes add the base to support the Vector data type for REST, GraphQL, and MCP. This PR focuses mainly on the REST endpoint, ensuring that it is able to both read and write to the vector columns through it. Lastly, it also adds support to ensure that OpenAPI is able to show columns with the Vector data type as array: [numeric].

The following files were changed to show that DAB is able to support the Vector data type and also change the type internally to system types it can work with. Such as the float type for DbType, the Numeric type for JsonDataType, and SINGLE_TYPE for GraphQL.

  • SqlTypeConstants
  • TypeHelper.cs
  • SqlSchemaConverter.cs

The following file was changed to allow DAB to save the vector columns as an array type, which is necessary for GraphQL to be able to know if the column of an entity is a simple type or an array.

  • MsSqlMetadataProvider.cs

The following files were changed to parse the information from the user to a format that can be used by DAB to create the queries for SQL.

  • BaseSqlQueryStructure.cs: Parses the array information by creating a new function that checks for the simple type of the array (e.g If it is an array of floats it will use the type float) and parse each value inside the array.
  • MsSqlQueryExecutor.cs: Parses the values given by the user and adds them to an SqlVector object which allows for an easier way to query the values.

The following file was changed to allow DAB's OpenAPI to support array types by checking if the main type is array and then taking the subtype and creating the appropriate item with it.

  • OpenApiDocumentor.cs

Note: All of these changes are only applicable to Vectors of Float32, support for Vectors of Float16 will have to wait as support for it in .NET is still missing.

How was this tested?

  • Integration Tests
  • Unit Tests

Added tests that ensure the following scenarios on REST:

  • Read regular vector
  • Read vector with max number of dimensions possible
  • Read vector with null value
  • Read vector with $first argument
  • Read vector with $after argument
  • Read vector with $filter argument - fails
  • Read vector with $orderby argument - fails
  • Insert regular vector
  • Insert vector with null value
  • Insert vector with number values in scientific format
  • Insert vector with number values in string format
  • Insert vector with more dimensions than allowed - fails
  • Insert vector with non-valid values - fails
  • Update vector with new values

Sample Request(s)

Note: Currently the values will be outputted in scientific notation, this will be changed in the future #3680

GET https://localhost:5001/api/dbo_normalvector/
image

POST https://localhost:5001/api/dbo_normalvector/

{
	"ProductID": 2001,
	"Embedding": [
		1.15202468461012559910005,
		0.00005,
		5.111111111
	]
}
image

PATCH https://localhost:5001/api/dbo_normalvector/ (Adding new row)

{
	"ProductID": 3000,
	"Embedding": [
		1.15202468461012559910005,
		0.00005,
		5.111111111
	]
}
image

PATCH https://localhost:5001/api/dbo_normalvector/ (Update existing row)

{
	"ProductID": 2001,
	"Embedding": [
		25,
		1.2,
		510.35
	]
}
image

PUT https://localhost:5001/api/dbo_normalvector/ (Adding new row)

{
	"ProductID": 4000,
	"Embedding": [
		0,
		1,
		37
	]
}
image

PUT https://localhost:5001/api/dbo_normalvector/ (Edit existing row)

{
	"ProductID": 2001,
	"Embedding": [
		0.220224962,
		0.66666667,
		37
	]
}
image

https://localhost:5001/swagger
image
image

souvikghosh04 and others added 25 commits June 10, 2026 10:51
Joint runtime + driver bump that prerequisites issue #2768 (MSSQL JSON
data type support). Behavior-preserving — no production logic changes.

WHY BOTH BUMPS ARE REQUIRED TOGETHER
====================================

SqlDbType.Json (numeric value 35) is a BCL enum value in System.Data,
added in .NET 9 and present in .NET 10. It is NOT a Microsoft.Data.SqlClient
symbol — SqlClient does not get to add values to a BCL enum it does not
own. On .NET 8, the SqlDbType enum stops at DateTimeOffset = 34, so

    Enum.TryParse<SqlDbType>("json", ignoreCase: true, out _)

in TypeHelper.GetSystemTypeFromSqlDbType returns false regardless of
which SqlClient version is installed. The companion feature PR for #2768
plans to add a single dictionary entry

    [SqlDbType.Json] = typeof(string)

which simply will not compile on net8.0. Both upgrades must therefore
land together as a single prerequisite.

WHY .NET 10 AND NOT .NET 9
==========================

- .NET 10 is the current Long-Term Support release (Nov 2025+).
- .NET 9 is Standard-Term Support, EOL May 2026.
- DAB's current .NET 8 line reaches EOL November 2026.

Going straight to LTS avoids a second forced upgrade inside 12 months.

SCOPE OF CHANGES (NO BEHAVIOR CHANGE INTENDED)
==============================================

Runtime / SDK pin:
  - global.json: 8.0.420 -> 10.0.301

Target framework (all 11 src/**/*.csproj):
  - net8.0 -> net10.0

NuGet packages (Directory.Packages.props):
  - Microsoft.Data.SqlClient                    5.2.3  -> 6.0.2
  - Microsoft.Extensions.Caching.Memory         8.0.1  -> 10.0.0
  - Microsoft.Extensions.Caching.Abstractions   9.0.0  -> 10.0.0
  - Microsoft.Extensions.Primitives             9.0.0  -> 10.0.0
  - Microsoft.Extensions.Configuration.Binder   9.0.0  -> 10.0.0
  - Microsoft.Extensions.Configuration.Json     9.0.0  -> 10.0.0
  - Microsoft.Extensions.Caching.StackExchangeRedis 9.0.3 -> 10.0.0

The Microsoft.Extensions.* bumps are required because OpenTelemetry's
transitive chain (Microsoft.Extensions.Logging.Configuration 10.0.0)
demands Configuration.Binder >= 10.0.0 — staying on 9.0.0 produces
NU1605 package-downgrade errors (warning-as-error).

Service.csproj (Microsoft.NET.Sdk.Web) — drop now-redundant
PackageReferences (NU1510, warning-as-error):
  - Microsoft.Extensions.Configuration.Binder  (in shared framework)
  - Microsoft.Extensions.Configuration.Json    (in shared framework)

Azure DevOps pipelines (9 occurrences across 8 files):
  - UseDotNet@2 version: 8.0.x -> 10.0.x

Container base images (Dockerfile):
  - mcr.microsoft.com/dotnet/sdk:8.0-cbl-mariner2.0   -> 10.0-azurelinux3.0
  - mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0 -> 10.0-azurelinux3.0
  (cbl-mariner2.0 tag does not exist for .NET 9+; Azure Linux 3.0 is
   Microsoft's documented successor.)

Aspire AppHost (src/Aspire.AppHost/AppHost.cs):
  - WithArgs("-f", "net8.0") -> WithArgs("-f", "net10.0")  (mssql + pg)

Build/publish scripts:
  - scripts/publish.ps1:               $dotnetTargetFrameworks net8.0 -> net10.0
  - scripts/create-manifest-file.ps1:  $dotnetTargetFrameworks + hashtable
                                       keys net8.0_{rid} -> net10.0_{rid}
                                       (TODO marker: release-engineering to
                                       confirm download URLs/hashes resolve)

License file rename:
  - external_licenses/Microsoft.Data.SqlClient.SNI.5.2.0.License.txt
    -> external_licenses/Microsoft.Data.SqlClient.SNI.6.0.0.License.txt
  - scripts/notice-generation.ps1: path updated to match
  - TODO marker at top of license file: contents still 5.2.0 text;
    release-engineering to refresh from upstream before merge.

SUPPRESSED ASPDEPR008 (warning-as-error) — DELIBERATE
=====================================================

ASP.NET Core 10 deprecates IWebHostBuilder / IWebHost in favor of
WebApplicationBuilder / IHost. Two locations in this repo still consume
the obsolete types and would block compilation under the repo's
TreatWarningsAsErrors=true setting:

  - src/Service/Program.cs (2 sites): test-only helpers
    CreateWebHostBuilder + CreateWebHostFromInMemoryUpdatableConfBuilder
    are consumed by the existing TestServer fixture which takes
    IWebHostBuilder.
  - src/Service.Tests/Configuration/ConfigurationTests.cs (3 sites):
    the consumer of those helpers.

Migrating from WebHost to HostBuilder/WebApplicationBuilder is a
behavior-affecting refactor and explicitly OUT OF SCOPE for this
no-behavior-change prerequisite PR. Suppression is added at the
project level (NoWarn=ASPDEPR008 on Service.csproj + Service.Tests.csproj)
with an inline TODO comment pointing at this branch and a follow-up
issue link placeholder.

VALIDATION
==========

dotnet --version          : 10.0.301
dotnet restore --nologo   : 11/11 projects restored (8.05s avg)
dotnet build  -c Debug    : 0 warnings, 0 errors, 10.32s

Multi-engine test categories (MsSql, PostgreSql, MySql, CosmosDb_NoSql,
DwSql) were NOT run locally; they MUST run green on the Azure DevOps
multi-engine matrix before this PR merges. That matrix is the gate.

NEXT STEPS BEFORE MERGE
=======================

1. Release-engineering: refresh
   external_licenses/Microsoft.Data.SqlClient.SNI.6.0.0.License.txt
   from upstream for the SNI 6.0.0 version (currently still 5.2.0 text
   with a TODO marker at top).
2. Release-engineering: confirm
   scripts/create-manifest-file.ps1 net10.0_{linux,win,osx}-x64 download
   URLs and SHA hashes resolve once the .NET 10 publish cycle runs.
3. Multi-engine CI matrix runs green (MsSql, PostgreSql, MySql,
   CosmosDb_NoSql, DwSql) — that is the merge gate.
4. File follow-up issue for ASPDEPR008 migration (WebHost ->
   WebApplicationBuilder / IHost) so the NoWarn suppressions can be
   removed in a future behavior-changing PR.
- Fix UseDotNet displayName v8.0.x -> v10.0.x across pipeline files

- Replace internal branch-path TODOs with durable follow-up notes

- Rename SNI license to 6.0.2 (actual runtime version) and fix notice-generation reference

- Bump Microsoft.AspNetCore.{TestHost,Authorization,Authentication.JwtBearer,Mvc.Testing} to 10.0.0
Aspire.Hosting pulls MessagePack 2.5.192 transitively (via KubernetesClient), which has high-severity advisory GHSA-hv8m-jj95-wg3x (CVE-2026-48109). The repo treats NU1903 as error, failing restore. Pin to patched 2.5.301 via CPM and add a direct reference in Aspire.AppHost to force the transitive upgrade.
…or change

SqlClient 6.x changed the internal SqlError 9-parameter constructor: the
win32ErrorCode parameter is now Int32 (was UInt32 in 5.x), and there are now
multiple 9-parameter overloads. Select the constructor by exact parameter
types and pass (int)0 for win32ErrorCode so CreateSqlException works across
SqlClient versions.
Starting with .NET 10, setting an environment variable to an empty string
preserves it as an empty value instead of deleting it (which previously
surfaced here as null). This caused ValidateAspNetCoreUrls to treat an empty
ASPNETCORE_URLS as invalid and exit with error, a regression from .NET 8 where
an empty value was equivalent to unset. Explicitly treat an empty value as
valid (Kestrel falls back to default URLs), preserving the prior behavior.
Whitespace-only values remain invalid.
These three files were modified by an earlier merge in a way unrelated to the
.NET 10 / SqlClient 6.x upgrade, reverting recent main formatting/cleanup:

- CLRtoJsonValueTypeUnitTests.cs: restore the DBTYPE_RESOLUTION_ERROR constant
  that was inadvertently removed.
- SqlUpdateQueryStructure.cs: restore main's formatting of the update-fields
  block.
- MutationEngineFactory.cs: restore main's version.

Restoring them to match main keeps this PR scoped to the upgrade and addresses
reviewer feedback (constant removal and formatting).
Base automatically changed from Usr/sogh/upgrade-net10-sqlclient6 to main June 24, 2026 17:30
@RubenCerna2079 RubenCerna2079 changed the title Dev/rubencerna/first changes vector type Add Support for Vector Data Type in SQL for REST Jun 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds SQL Server vector(N) column support in DAB’s SQL layer so REST endpoints can read/write vectors as JSON number arrays, with accompanying integration tests and metadata/type mappings.

Changes:

  • Adds REST integration tests and MSSQL schema/config updates for a VectorType table/entity.
  • Extends MSSQL metadata/type handling to recognize vector columns and treat them as array-like for downstream processing.
  • Adds parameter parsing/execution support to accept JSON arrays and pass them to SqlClient as SqlVector<float>.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/Service.Tests/SqlTests/RestApiTests/MsSqlRestVectorTypesTests.cs New REST integration tests covering vector read/write and pagination behavior.
src/Service.Tests/DatabaseSchema-MsSql.sql Adds vector_type_table and seed data, including a max-dimension vector row.
src/Service.Tests/dab-config.MsSql.json Adds VectorType entity configuration and updates the MSSQL connection string.
src/Service.GraphQLBuilder/Sql/SchemaConverter.cs Adds GraphQL type mapping for SqlVector<T>.
src/Core/Services/TypeHelper.cs Adds DbType/JsonDataType/SqlDbType mappings related to vector support.
src/Core/Services/MetadataProviders/MsSqlMetadataProvider.cs Detects vector columns and marks them as array-like for metadata consumers.
src/Core/Resolvers/Sql Query Structures/BaseSqlQueryStructure.cs Adds parsing logic for Single[] values from JSON arrays.
src/Core/Resolvers/MsSqlQueryExecutor.cs Converts vector parameters to SqlVector<float> before sending to SQL Server.
src/Core/Resolvers/MsSqlQueryBuilder.cs Removes vector from the “unsupported types” list so it can be handled.
src/Core/Models/SqlTypeConstants.cs Marks vector as a supported SQL type literal.
src/Config/Azure.DataApiBuilder.Config.csproj File encoding change (BOM added).
config-generators/mssql-commands.txt Adds generator commands for the new VectorType entity.

Comment thread src/Service.Tests/dab-config.MsSql.json Outdated
Comment thread src/Core/Resolvers/MsSqlQueryExecutor.cs
Comment thread src/Core/Resolvers/Sql Query Structures/BaseSqlQueryStructure.cs
Comment thread src/Core/Services/MetadataProviders/MsSqlMetadataProvider.cs
Comment thread src/Service.Tests/DatabaseSchema-MsSql.sql
Comment thread src/Config/Azure.DataApiBuilder.Config.csproj Outdated
RubenCerna2079 and others added 6 commits June 25, 2026 10:30
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…t-changes-vector-type' into dev/rubencerna/first-changes-vector-type
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Foundation for adding support for Vector Data Type

6 participants