Add Spark 4.0 Driver API coverage and .NET Interactive support

[SparkSnail]

Summary

Adds Spark 4.0 compatibility coverage for existing Driver APIs and fixes .NET Interactive cross-cell UDF execution.

Changes

• Register Interactive assemblies and dependency files in the querying session’s artifact scope on Spark 4.0. Previously, files registered through bare SparkContext.AddFile could be visible on the Driver but unavailable to SQL workers. Spark 3 retains its existing publishing path.
• Avoid republishing the previous submission after a compilation failure, while preserving newly compiled submissions after runtime failures.
• Allow Spark 4.0 paths containing spaces and support CSharpKernel instances without the optional NuGet provider.
• Add result/schema assertions for SQL, DataFrame transformations, Parquet round trips, Catalog metadata, and ANSI behavior. Tighten DataFrameWriterV2 negative assertions.
• Add DriverApi tests to the Spark 4 focused CI suite and dedicated Interactive E2E steps for Spark 3.5.3 and Spark 4.0.x.

Validation

Local Windows validation using official Spark distributions:

• Spark 4.0.4 and Spark 3.5.3: DriverApi tests passed, 38/38 each.
• Spark 4.0.4 and Spark 3.5.3: Interactive E2E passed, 1/1 each. Each scenario covers multiple cells, newly defined types, runtime failures, compilation failures, and recovery.
• Interactive unit tests: 15/15 passed.
• Spark 4 SQLUtils tests: 6/6 passed.
• Spark 4 expanded regression suite: 92/92 passed.
• C# build, YAML parsing, and git diff --check passed.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).