feat(blob): blob versioning support, opt-in per storage account (#665)

ChangeLog.md

@@ -4,6 +4,17 @@
 
 ## Upcoming Release
 
+Blob:
+
+- Added support for blob versioning on the LokiJS metadata store, opt in per storage account (issue #665). Versioning is an ARM management plane setting in Azure Storage, not part of the data plane REST API that Azurite emulates, so it is supplied at start up with the new mutually exclusive `--accountConfigFile <path>` and `--accountConfig <json>` options (and the matching `azurite.accountConfigFile` / `azurite.accountConfig` VS Code settings) rather than through a data plane API. Multiple accounts are supported, and an account that is not listed keeps the default of versioning disabled. See `docs/designs/blob-versioning.md`.
+  - Version creating operations: `Put Blob`, `Put Block List`, `Copy Blob`, `Set Blob Metadata` (all blob types), `Page Blob Create`, `Append Blob Create`, and `Snapshot Blob` create a version and return `x-ms-version-id`. `Put Page`, `Append Block` and `Set Blob Properties` do not create a version, matching the service.
+  - `Get Blob` and `Get Blob Properties` accept `?versionid=`, and return `x-ms-is-current-version` only for the current version. `List Blobs` supports `include=versions`. `Get Blob Tags`, `Set Blob Tags` and `Set Blob Tier` accept `?versionid=`, so tags and access tier are addressable per version. A version is restored by copying it over the current version.
+  - `Delete Blob` without a version ID turns the current version into a previous version and retains it, leaving the blob with no current version while previous versions persist. `Delete Blob` with `?versionid=` removes a single previous version; naming the current version returns 403 `OperationNotAllowedOnRootBlob`. Combining `?snapshot=` with `?versionid=` returns 400 `MutuallyExclusiveQueryParameters`, and a malformed `?versionid=` returns 400 `InvalidQueryParameterValue`. In loose mode an unusable `versionid` is ignored rather than rejected.
+  - Versioning can be turned on and off against an existing workspace, as on a real account. Turning it off keeps existing versions listed, readable and deletable by version ID, and a later write retains the previously current version while producing a blob that is not itself a version.
+  - Not implemented: the SQL metadata store (configuring versioning with `AZURITE_DB` fails at start up), blob version SAS (`sr=bv` and the `x` permission), soft delete interactions, blob expiration, and `HasVersionsOnly`, which the service reports only under `include=deletedwithversions`.
+- Moved account (management plane) configuration into a standalone `src/common/account/` module with its own database file (`__azurite_db_account__.json`), so that the queue and table services can share it rather than reading configuration owned by the blob service.
+- Blob list continuation tokens now carry a secondary key so that a `List Blobs` page can resume part way through one blob's versions. Tokens for listings that do not involve versions keep the previous plain blob name format, so existing tokens remain valid.
+
 General:
 
 - Raised the minimum supported Node.js runtime from 21 to 22 because Node.js 21 has reached end of life.

README.md

@@ -208,6 +208,8 @@ Following extension configurations are supported:
 - `azurite.inMemoryPersistence` Disable persisting any data to disk. If the Azurite process is terminated, all data is lost.
 - `azurite.extentMemoryLimit` When using in-memory persistence, limit the total size of extents (blob and queue content) to a specific number of megabytes. This does not limit blob, queue, or table metadata. Defaults to 50% of total memory.
 - `azurite.disableTelemetry` Disable telemetry data collection of this Azurite execution. By default, Azurite will collect telemetry data to help improve the product.
+- `azurite.accountConfigFile` Path to a JSON file with account level (management plane) configuration, for example to enable blob versioning. Mutually exclusive with `azurite.accountConfig`.
+- `azurite.accountConfig` Inline JSON string with account level (management plane) configuration, for example to enable blob versioning. Mutually exclusive with `azurite.accountConfigFile`.
 
 ### [DockerHub](https://hub.docker.com/_/microsoft-azure-storage-azurite)
 
@@ -240,7 +242,7 @@ docker run -p 10000:10000 -p 10001:10001 -v c:/azurite:/data mcr.microsoft.com/a
 #### Customize all Azurite V3 supported parameters for docker image
 
 ```bash
-docker run -p 7777:7777 -p 8888:8888 -p 9999:9999 -v c:/azurite:/workspace mcr.microsoft.com/azure-storage/azurite azurite -l /workspace -d /workspace/debug.log --blobPort 7777 --blobHost 0.0.0.0 --blobKeepAliveTimeout 5 --queuePort 8888 --queueHost 0.0.0.0 --queueKeepAliveTimeout 5 --tablePort 9999 --tableHost 0.0.0.0 --tableKeepAliveTimeout 5 --loose --skipApiVersionCheck --disableProductStyleUrl --disableTelemetry
+docker run -p 7777:7777 -p 8888:8888 -p 9999:9999 -v c:/azurite:/workspace mcr.microsoft.com/azure-storage/azurite azurite -l /workspace -d /workspace/debug.log --blobPort 7777 --blobHost 0.0.0.0 --blobKeepAliveTimeout 5 --queuePort 8888 --queueHost 0.0.0.0 --queueKeepAliveTimeout 5 --tablePort 9999 --tableHost 0.0.0.0 --tableKeepAliveTimeout 5 --loose --skipApiVersionCheck --disableProductStyleUrl --disableTelemetry --accountConfigFile /workspace/myAccountConfig.json
 ```
 
 Above command will try to start Azurite image with configurations:
@@ -275,6 +277,8 @@ Above command will try to start Azurite image with configurations:
 
 `--azurite.disableTelemetry` disable telemetry data collection of this Azurite execution. By default, Azurite will collect telemetry data to help improve the product.
 
+`--accountConfigFile` / `--accountConfig` supply account level (management plane) configuration such as enabling blob versioning. See [Blob versioning](#blob-versioning).
+
 > If you use customized azurite parameters for docker image, `--blobHost 0.0.0.0`, `--queueHost 0.0.0.0` are required parameters.
 
 > In above sample, you need to use **double first forward slash** for location and debug path parameters to avoid a [known issue](https://stackoverflow.com/questions/48427366/docker-build-command-add-c-program-files-git-to-the-path-passed-as-build-argu) for Git on Windows.
@@ -458,6 +462,78 @@ Optional. By default, Azurite will collect telemetry data to help improve the pr
 --disableTelemetry
 ```
 
+### Blob versioning
+
+Optional. Enable [blob versioning](https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-overview)
+for one or more storage accounts. When versioning is enabled for an account, overwriting
+a blob preserves the previous content as a version addressable by its version ID.
+
+In Azure Storage, versioning is an account level setting configured through the ARM
+management plane (`Microsoft.Storage/storageAccounts/blobServices/default`, property
+`isVersioningEnabled`), not through the data plane REST API that Azurite emulates.
+Azurite has no management plane, so the setting is supplied at start up with one of two
+mutually exclusive options - a JSON file, or an inline JSON string:
+
+```cmd
+--accountConfigFile <path>
+```
+
+```cmd
+--accountConfig <json>
+```
+
+For example, with `./myAccountConfig.json`:
+
+```json
+{
+  "accounts": [
+    {
+      "name": "devstoreaccount1",
+      "blobService": {
+        "isVersioningEnabled": true
+      }
+    }
+  ]
+}
+```
+
+```cmd
+azurite --accountConfigFile ./myAccountConfig.json
+```
+
+Or inline, without a file:
+
+```cmd
+azurite --accountConfig "{\"accounts\":[{\"name\":\"devstoreaccount1\",\"blobService\":{\"isVersioningEnabled\":true}}]}"
+```
+
+A single account can also be given directly, without the `accounts` wrapper:
+
+```json
+{ "name": "devstoreaccount1", "blobService": { "isVersioningEnabled": true } }
+```
+
+Account names are matched case insensitively, and any account not listed keeps the
+default of versioning disabled - so this configuration only opts accounts in, it never
+changes behaviour for accounts you did not name. If you run Azurite with
+[multiple accounts](#customized-storage-accounts--keys), list each account that needs
+versioning; a single configuration is not applied to all of them implicitly. The resolved
+configuration is written to the debug log at start up.
+
+Versioning can be turned on and off freely, as on a real storage account. The setting is
+persisted alongside the metadata, so starting Azurite against an existing workspace
+without these options keeps whatever was configured last time. Turning versioning off
+keeps existing versions readable by version ID; later writes simply stop creating new
+ones.
+
+Versioning is implemented for the default LokiJS metadata store only. Configuring it
+together with the SQL based metadata implementation (via `AZURITE_DB`) is rejected at
+start up.
+
+See [docs/designs/blob-versioning.md](docs/designs/blob-versioning.md) for the data model,
+the supported operations, and the interactions that are not implemented (blob version
+SAS, soft delete, blob expiration).
+
 ### Use in-memory storage
 
 Optional. Disable persisting any data to disk and only store data in-memory. If the Azurite process is terminated, all
@@ -1073,6 +1149,7 @@ Detailed support matrix:
   - Copy Blob (Only supports copy within same Azurite instance)
   - Abort Copy Blob (Only supports copy within same Azurite instance)
   - Copy Blob From URL (Only supports copy within same Azurite instance, only on Loki)
+  - Blob Versions (Only on Loki, opt in per account, see [Blob versioning](#blob-versioning))
   - Access control based on conditional headers
 - Following features or REST APIs are NOT supported or limited supported in this release (will support more features per customers feedback in future releases)
   - SharedKey Lite
@@ -1081,7 +1158,7 @@ Detailed support matrix:
   - Soft delete & Undelete Blob
   - Incremental Copy Blob
   - Blob Query
-  - Blob Versions
+  - Blob Version SAS (the `x` permission and the `sr=bv` signed resource)
   - Blob Last Access Time
   - Concurrent Append
   - Blob Expiry