Add Support for Vector Data Type in SQL for REST#3677
Open
RubenCerna2079 wants to merge 37 commits into
Open
Conversation
Joint runtime + driver bump that prerequisites issue #2768 (MSSQL JSON data type support). Behavior-preserving — no production logic changes. WHY BOTH BUMPS ARE REQUIRED TOGETHER ==================================== SqlDbType.Json (numeric value 35) is a BCL enum value in System.Data, added in .NET 9 and present in .NET 10. It is NOT a Microsoft.Data.SqlClient symbol — SqlClient does not get to add values to a BCL enum it does not own. On .NET 8, the SqlDbType enum stops at DateTimeOffset = 34, so Enum.TryParse<SqlDbType>("json", ignoreCase: true, out _) in TypeHelper.GetSystemTypeFromSqlDbType returns false regardless of which SqlClient version is installed. The companion feature PR for #2768 plans to add a single dictionary entry [SqlDbType.Json] = typeof(string) which simply will not compile on net8.0. Both upgrades must therefore land together as a single prerequisite. WHY .NET 10 AND NOT .NET 9 ========================== - .NET 10 is the current Long-Term Support release (Nov 2025+). - .NET 9 is Standard-Term Support, EOL May 2026. - DAB's current .NET 8 line reaches EOL November 2026. Going straight to LTS avoids a second forced upgrade inside 12 months. SCOPE OF CHANGES (NO BEHAVIOR CHANGE INTENDED) ============================================== Runtime / SDK pin: - global.json: 8.0.420 -> 10.0.301 Target framework (all 11 src/**/*.csproj): - net8.0 -> net10.0 NuGet packages (Directory.Packages.props): - Microsoft.Data.SqlClient 5.2.3 -> 6.0.2 - Microsoft.Extensions.Caching.Memory 8.0.1 -> 10.0.0 - Microsoft.Extensions.Caching.Abstractions 9.0.0 -> 10.0.0 - Microsoft.Extensions.Primitives 9.0.0 -> 10.0.0 - Microsoft.Extensions.Configuration.Binder 9.0.0 -> 10.0.0 - Microsoft.Extensions.Configuration.Json 9.0.0 -> 10.0.0 - Microsoft.Extensions.Caching.StackExchangeRedis 9.0.3 -> 10.0.0 The Microsoft.Extensions.* bumps are required because OpenTelemetry's transitive chain (Microsoft.Extensions.Logging.Configuration 10.0.0) demands Configuration.Binder >= 10.0.0 — staying on 9.0.0 produces NU1605 package-downgrade errors (warning-as-error). Service.csproj (Microsoft.NET.Sdk.Web) — drop now-redundant PackageReferences (NU1510, warning-as-error): - Microsoft.Extensions.Configuration.Binder (in shared framework) - Microsoft.Extensions.Configuration.Json (in shared framework) Azure DevOps pipelines (9 occurrences across 8 files): - UseDotNet@2 version: 8.0.x -> 10.0.x Container base images (Dockerfile): - mcr.microsoft.com/dotnet/sdk:8.0-cbl-mariner2.0 -> 10.0-azurelinux3.0 - mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0 -> 10.0-azurelinux3.0 (cbl-mariner2.0 tag does not exist for .NET 9+; Azure Linux 3.0 is Microsoft's documented successor.) Aspire AppHost (src/Aspire.AppHost/AppHost.cs): - WithArgs("-f", "net8.0") -> WithArgs("-f", "net10.0") (mssql + pg) Build/publish scripts: - scripts/publish.ps1: $dotnetTargetFrameworks net8.0 -> net10.0 - scripts/create-manifest-file.ps1: $dotnetTargetFrameworks + hashtable keys net8.0_{rid} -> net10.0_{rid} (TODO marker: release-engineering to confirm download URLs/hashes resolve) License file rename: - external_licenses/Microsoft.Data.SqlClient.SNI.5.2.0.License.txt -> external_licenses/Microsoft.Data.SqlClient.SNI.6.0.0.License.txt - scripts/notice-generation.ps1: path updated to match - TODO marker at top of license file: contents still 5.2.0 text; release-engineering to refresh from upstream before merge. SUPPRESSED ASPDEPR008 (warning-as-error) — DELIBERATE ===================================================== ASP.NET Core 10 deprecates IWebHostBuilder / IWebHost in favor of WebApplicationBuilder / IHost. Two locations in this repo still consume the obsolete types and would block compilation under the repo's TreatWarningsAsErrors=true setting: - src/Service/Program.cs (2 sites): test-only helpers CreateWebHostBuilder + CreateWebHostFromInMemoryUpdatableConfBuilder are consumed by the existing TestServer fixture which takes IWebHostBuilder. - src/Service.Tests/Configuration/ConfigurationTests.cs (3 sites): the consumer of those helpers. Migrating from WebHost to HostBuilder/WebApplicationBuilder is a behavior-affecting refactor and explicitly OUT OF SCOPE for this no-behavior-change prerequisite PR. Suppression is added at the project level (NoWarn=ASPDEPR008 on Service.csproj + Service.Tests.csproj) with an inline TODO comment pointing at this branch and a follow-up issue link placeholder. VALIDATION ========== dotnet --version : 10.0.301 dotnet restore --nologo : 11/11 projects restored (8.05s avg) dotnet build -c Debug : 0 warnings, 0 errors, 10.32s Multi-engine test categories (MsSql, PostgreSql, MySql, CosmosDb_NoSql, DwSql) were NOT run locally; they MUST run green on the Azure DevOps multi-engine matrix before this PR merges. That matrix is the gate. NEXT STEPS BEFORE MERGE ======================= 1. Release-engineering: refresh external_licenses/Microsoft.Data.SqlClient.SNI.6.0.0.License.txt from upstream for the SNI 6.0.0 version (currently still 5.2.0 text with a TODO marker at top). 2. Release-engineering: confirm scripts/create-manifest-file.ps1 net10.0_{linux,win,osx}-x64 download URLs and SHA hashes resolve once the .NET 10 publish cycle runs. 3. Multi-engine CI matrix runs green (MsSql, PostgreSql, MySql, CosmosDb_NoSql, DwSql) — that is the merge gate. 4. File follow-up issue for ASPDEPR008 migration (WebHost -> WebApplicationBuilder / IHost) so the NoWarn suppressions can be removed in a future behavior-changing PR.
- Fix UseDotNet displayName v8.0.x -> v10.0.x across pipeline files
- Replace internal branch-path TODOs with durable follow-up notes
- Rename SNI license to 6.0.2 (actual runtime version) and fix notice-generation reference
- Bump Microsoft.AspNetCore.{TestHost,Authorization,Authentication.JwtBearer,Mvc.Testing} to 10.0.0
Aspire.Hosting pulls MessagePack 2.5.192 transitively (via KubernetesClient), which has high-severity advisory GHSA-hv8m-jj95-wg3x (CVE-2026-48109). The repo treats NU1903 as error, failing restore. Pin to patched 2.5.301 via CPM and add a direct reference in Aspire.AppHost to force the transitive upgrade.
…or change SqlClient 6.x changed the internal SqlError 9-parameter constructor: the win32ErrorCode parameter is now Int32 (was UInt32 in 5.x), and there are now multiple 9-parameter overloads. Select the constructor by exact parameter types and pass (int)0 for win32ErrorCode so CreateSqlException works across SqlClient versions.
Starting with .NET 10, setting an environment variable to an empty string preserves it as an empty value instead of deleting it (which previously surfaced here as null). This caused ValidateAspNetCoreUrls to treat an empty ASPNETCORE_URLS as invalid and exit with error, a regression from .NET 8 where an empty value was equivalent to unset. Explicitly treat an empty value as valid (Kestrel falls back to default URLs), preserving the prior behavior. Whitespace-only values remain invalid.
These three files were modified by an earlier merge in a way unrelated to the .NET 10 / SqlClient 6.x upgrade, reverting recent main formatting/cleanup: - CLRtoJsonValueTypeUnitTests.cs: restore the DBTYPE_RESOLUTION_ERROR constant that was inadvertently removed. - SqlUpdateQueryStructure.cs: restore main's formatting of the update-fields block. - MutationEngineFactory.cs: restore main's version. Restoring them to match main keeps this PR scoped to the upgrade and addresses reviewer feedback (constant removal and formatting).
Contributor
There was a problem hiding this comment.
Pull request overview
Adds SQL Server vector(N) column support in DAB’s SQL layer so REST endpoints can read/write vectors as JSON number arrays, with accompanying integration tests and metadata/type mappings.
Changes:
- Adds REST integration tests and MSSQL schema/config updates for a
VectorTypetable/entity. - Extends MSSQL metadata/type handling to recognize vector columns and treat them as array-like for downstream processing.
- Adds parameter parsing/execution support to accept JSON arrays and pass them to SqlClient as
SqlVector<float>.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Service.Tests/SqlTests/RestApiTests/MsSqlRestVectorTypesTests.cs | New REST integration tests covering vector read/write and pagination behavior. |
| src/Service.Tests/DatabaseSchema-MsSql.sql | Adds vector_type_table and seed data, including a max-dimension vector row. |
| src/Service.Tests/dab-config.MsSql.json | Adds VectorType entity configuration and updates the MSSQL connection string. |
| src/Service.GraphQLBuilder/Sql/SchemaConverter.cs | Adds GraphQL type mapping for SqlVector<T>. |
| src/Core/Services/TypeHelper.cs | Adds DbType/JsonDataType/SqlDbType mappings related to vector support. |
| src/Core/Services/MetadataProviders/MsSqlMetadataProvider.cs | Detects vector columns and marks them as array-like for metadata consumers. |
| src/Core/Resolvers/Sql Query Structures/BaseSqlQueryStructure.cs | Adds parsing logic for Single[] values from JSON arrays. |
| src/Core/Resolvers/MsSqlQueryExecutor.cs | Converts vector parameters to SqlVector<float> before sending to SQL Server. |
| src/Core/Resolvers/MsSqlQueryBuilder.cs | Removes vector from the “unsupported types” list so it can be handled. |
| src/Core/Models/SqlTypeConstants.cs | Marks vector as a supported SQL type literal. |
| src/Config/Azure.DataApiBuilder.Config.csproj | File encoding change (BOM added). |
| config-generators/mssql-commands.txt | Adds generator commands for the new VectorType entity. |
…t-changes-vector-type' into dev/rubencerna/first-changes-vector-type
Aniruddh25
reviewed
Jun 26, 2026
Aniruddh25
reviewed
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why make this change?
What is this change?
These changes add the base to support the Vector data type for REST, GraphQL, and MCP. This PR focuses mainly on the REST endpoint, ensuring that it is able to both read and write to the vector columns through it. Lastly, it also adds support to ensure that OpenAPI is able to show columns with the Vector data type as
array: [numeric].The following files were changed to show that DAB is able to support the Vector data type and also change the type internally to system types it can work with. Such as the
floattype forDbType, theNumerictype forJsonDataType, andSINGLE_TYPEfor GraphQL.The following file was changed to allow DAB to save the vector columns as an array type, which is necessary for GraphQL to be able to know if the column of an entity is a simple type or an array.
The following files were changed to parse the information from the user to a format that can be used by DAB to create the queries for SQL.
The following file was changed to allow DAB's OpenAPI to support array types by checking if the main type is
arrayand then taking the subtype and creating the appropriate item with it.Note: All of these changes are only applicable to Vectors of Float32, support for Vectors of Float16 will have to wait as support for it in .NET is still missing.
How was this tested?
Added tests that ensure the following scenarios on REST:
$firstargument$afterargument$filterargument - fails$orderbyargument - failsSample Request(s)
Note: Currently the values will be outputted in scientific notation, this will be changed in the future #3680
GET https://localhost:5001/api/dbo_normalvector/

POST https://localhost:5001/api/dbo_normalvector/
PATCH https://localhost:5001/api/dbo_normalvector/ (Adding new row)
PATCH https://localhost:5001/api/dbo_normalvector/ (Update existing row)
PUT https://localhost:5001/api/dbo_normalvector/ (Adding new row)
PUT https://localhost:5001/api/dbo_normalvector/ (Edit existing row)
https://localhost:5001/swagger

