feat(dpp)!: typed scalar arrays in document schemas (PV14)

book/src/data-model/data-contracts.md

@@ -243,6 +243,35 @@ What happens to data:
 - **New writes are held to the new schema.** Creates must supply the property; replaces re-supply full content, so replacing a grandfathered document requires the new property and re-stamps the document at the current version — lazy migration, one document at a time.
 - **Indexes are unaffected** because index additions on update remain banned — a newly added required field cannot be indexed retroactively (there is no backfill).
 
+## Typed Scalar Arrays
+
+Until protocol v14 the only `type: array` a document schema could declare was a byte array (`byteArray: true`). From v14 (meta-schema v3) an array property may instead declare an `items` schema, which makes it a **typed scalar array**: a list of one scalar type stored inline in the document, exactly like every other property.
+
+```json
+"reasons": {
+  "type": "array",
+  "minItems": 0,
+  "maxItems": 64,
+  "uniqueItems": true,
+  "items": {
+    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
+    "contentMediaType": "application/x.dash.dpp.identifier"
+  },
+  "position": 2
+}
+```
+
+The rules, enforced by the meta-schema on the validating path and by the parser (`parse_typed_array` version 0) on both paths, except the bound, which is a registration limit:
+
+- `items` is any scalar property schema the parser already reads at the top level: an integer, a number, a string with `minLength` / `maxLength`, a boolean, a byte array with `minItems` / `maxItems`, or an identifier. Arrays of objects, arrays of arrays, a `$ref`, `enum` / `const`, and `refersTo` on the items are refused (a reference on identifier items is a separate follow-up; the item schema is parsed whole so it can carry one later).
+- `minItems` and `maxItems` on the array count elements, not bytes. Registration (full validation, in the generation 3 driver) requires `maxItems` and refuses one above `SystemLimits::max_typed_array_items` (1024), because the count-prefixed inline encoding is sized for fees by its bound; like every registration limit it is not re-applied when a stored contract is read. `uniqueItems: true` refuses a repeated element.
+- An array is either a byte array or a typed array, never both: `byteArray` and `items` are mutually exclusive, and `contentMediaType` belongs on the items.
+- An array property cannot be indexed. Drive's query conditions give the type no operator, so an index on one is refused at registration with `InvalidIndexPropertyTypeError`, and an indexOnly `entryPayload` cannot name one. A `propertyAgreement` cannot bind one either: the write-time check compares index key encodings, so drive-abci refuses the declaration at registration.
+
+The parsed form is `DocumentPropertyType::TypedArray(TypedArrayProperty { items, min_items, max_items, unique_items })`, with `items` an `ArrayItemType`, produced by the versioned `parse_typed_array` method that runs before the scalar parser (`None` before protocol v14, so the scalar parser's byte-array-only refusal stays byte-identical there). Documents carry the list as a `Value::Array` of the item type's values, validated by the JSON schema validator (`minItems`, `maxItems`, `uniqueItems`, and the item constraints all surface as the usual `JsonSchemaError`), and serialized as a varint element count followed by the elements (see [Document Serialization](../serialization/document-serialization.md)). Fee estimation sizes the property by `maxItems` times the item bound plus the count prefix. Identifier and byte array items are registered as `path[]` conversion paths, so a document built from JSON converts every element of the list.
+
+The Swift and Kotlin contract parsers still read every `type: array` as a byte array; typed arrays reach those SDKs in a follow-up.
+
 ## Rules and Guidelines
 
 **Do:**

book/src/serialization/document-serialization.md

@@ -184,7 +184,7 @@ All numeric values use **big-endian** byte order.
 | `byteArray` (variable size) | varint length prefix + raw bytes |
 | `identifier` | 32 bytes raw |
 | `date` | 8 bytes big-endian f64 (when optional: `0xff` prefix + 8 bytes) |
-| `array` | varint element count + each element encoded in sequence |
+| `array` (typed scalar array, protocol v14) | varint element count + each element in sequence: integer 8 bytes, number 8 bytes, boolean 1 byte, string / byte array / identifier varint length prefix + bytes (an identifier item is always 33 bytes) |
 | `object` | Nested fields serialized recursively in their schema position order |
 
 **Note on date types**: User-property `date` fields are encoded as **f64** (8 bytes). System timestamps (`$createdAt`, `$updatedAt`, `$transferredAt`) are **u64** milliseconds. Both are 8 bytes big-endian but use different numeric representations.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://github.com/dashpay/platform/blob/master/packages/rs-dpp/schema/meta_schemas/document/v1/document-meta.json",
-  "$comment": "EDITABLE UNTIL 4.2 (PROTOCOL V14) IS LIVE ON MAINNET; FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties, the requiredSince property keyword (the contract version a property is required from), and the timeRange index transform, and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once 4.2 is live on mainnet, mutating it would change historical validation results and break consensus replay, and any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
+  "$comment": "EDITABLE UNTIL 4.2 (PROTOCOL V14) IS LIVE ON MAINNET; FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties, the requiredSince property keyword (the contract version a property is required from), the timeRange index transform, and typed scalar arrays (an array property declared by an items schema instead of byteArray, stored inline as a count followed by the elements), and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once 4.2 is live on mainnet, mutating it would change historical validation results and break consensus replay, and any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
   "type": "object",
   "$defs": {
     "documentProperties": {
@@ -36,6 +36,155 @@
         "unevaluatedProperties": false
       }
     },
+    "typedArrayItemSchema": {
+      "$comment": "The items schema of a typed scalar array: one scalar type with its bounds, read by the property parser into the array's item type. Objects, arrays that are not byte arrays (arrays of arrays), $ref, enum, const, refersTo (a reference on identifier items is a separate follow-up) and position are not items",
+      "type": "object",
+      "properties": {
+        "type": {
+          "enum": [
+            "string",
+            "integer",
+            "number",
+            "boolean",
+            "array"
+          ]
+        },
+        "$comment": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/core#/properties/$comment"
+        },
+        "description": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/meta-data#/properties/description"
+        },
+        "multipleOf": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/multipleOf"
+        },
+        "maximum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/maximum"
+        },
+        "exclusiveMaximum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/exclusiveMaximum"
+        },
+        "minimum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/minimum"
+        },
+        "exclusiveMinimum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/exclusiveMinimum"
+        },
+        "maxLength": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/maxLength"
+        },
+        "minLength": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/minLength"
+        },
+        "pattern": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/pattern"
+        },
+        "format": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/format-annotation#/properties/format"
+        },
+        "maxItems": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/maxItems"
+        },
+        "minItems": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/minItems"
+        },
+        "contentMediaType": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/content#/properties/contentMediaType"
+        },
+        "byteArray": {
+          "type": "boolean",
+          "const": true
+        }
+      },
+      "required": [
+        "type"
+      ],
+      "additionalProperties": false,
+      "dependentSchemas": {
+        "byteArray": {
+          "description": "should be used only with array type",
+          "properties": {
+            "type": {
+              "type": "string",
+              "const": "array"
+            }
+          }
+        },
+        "contentMediaType": {
+          "if": {
+            "properties": {
+              "contentMediaType": {
+                "const": "application/x.dash.dpp.identifier"
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "byteArray": {
+                "const": true
+              },
+              "minItems": {
+                "const": 32
+              },
+              "maxItems": {
+                "const": 32
+              }
+            },
+            "required": [
+              "byteArray",
+              "minItems",
+              "maxItems"
+            ]
+          }
+        },
+        "pattern": {
+          "description": "prevent slow pattern matching of large strings",
+          "properties": {
+            "maxLength": {
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 50000
+            }
+          },
+          "required": [
+            "maxLength"
+          ]
+        },
+        "format": {
+          "description": "prevent slow format validation of large strings",
+          "properties": {
+            "maxLength": {
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 50000
+            }
+          },
+          "required": [
+            "maxLength"
+          ]
+        }
+      },
+      "allOf": [
+        {
+          "$comment": "an array item is a byte array: arrays of arrays are not supported",
+          "if": {
+            "properties": {
+              "type": {
+                "const": "array"
+              }
+            },
+            "required": [
+              "type"
+            ]
+          },
+          "then": {
+            "required": [
+              "byteArray"
+            ]
+          }
+        }
+      ]
+    },
     "documentSchema": {
       "type": "object",
       "properties": {
@@ -91,6 +240,10 @@
         "uniqueItems": {
           "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/uniqueItems"
         },
+        "items": {
+          "description": "The item schema of a typed scalar array (protocol version 14): one scalar type with its bounds. The array is stored inline in the document as a count followed by the elements; minItems and maxItems on the array count elements, uniqueItems refuses a repeated element. Only allowed on an array property that is not a byte array",
+          "$ref": "#/$defs/typedArrayItemSchema"
+        },
         "refersTo": {
           "type": "object",
           "properties": {
@@ -327,6 +480,18 @@
             }
           }
         },
+        "items": {
+          "description": "should be used only with array type",
+          "properties": {
+            "type": {
+              "type": "string",
+              "const": "array"
+            }
+          },
+          "required": [
+            "type"
+          ]
+        },
         "contentMediaType": {
           "if": {
             "properties": {
@@ -439,7 +604,7 @@
           }
         },
         {
-          "$comment": "allow only byte arrays",
+          "$comment": "an array property is either a byte array (byteArray: true, no items) or a typed scalar array (an items schema and maxItems, no byteArray). Typed scalar arrays exist from protocol version 14: minItems and maxItems count elements, uniqueItems refuses a repeated element, and contentMediaType belongs on the items",
           "if": {
             "properties": {
               "type": {
@@ -451,11 +616,30 @@
             ]
           },
           "then": {
-            "properties": {
-              "byteArray": true
-            },
-            "required": [
-              "byteArray"
+            "oneOf": [
+              {
+                "$comment": "byte array",
+                "properties": {
+                  "byteArray": true,
+                  "items": false
+                },
+                "required": [
+                  "byteArray"
+                ]
+              },
+              {
+                "$comment": "typed scalar array",
+                "properties": {
+                  "items": true,
+                  "maxItems": true,
+                  "byteArray": false,
+                  "contentMediaType": false
+                },
+                "required": [
+                  "items",
+                  "maxItems"
+                ]
+              }
             ]
           }
         },

packages/rs-dpp/src/data_contract/document_type/class_methods/mod.rs

@@ -8,6 +8,7 @@ use crate::ProtocolError;
 
 pub(crate) mod apply_required_since;
 mod create_document_types_from_document_schemas;
+pub(crate) mod parse_typed_array;
 mod should_use_creator_id;
 mod system_properties;
 mod try_from_schema;

packages/rs-dpp/src/data_contract/document_type/class_methods/parse_typed_array/mod.rs

@@ -0,0 +1,80 @@
+use crate::data_contract::document_type::DocumentPropertyType;
+use crate::data_contract::errors::DataContractError;
+use platform_value::Value;
+use platform_version::version::PlatformVersion;
+use std::collections::BTreeMap;
+
+mod v0;
+
+/// Parses a typed scalar array property, `type: array` with an `items` schema
+/// in place of `byteArray: true`, into [`DocumentPropertyType::TypedArray`].
+///
+/// Returns `None` for every other property, a byte array included, which the
+/// caller leaves to `DocumentPropertyType::try_from_value_map` exactly as
+/// before typed arrays existed.
+///
+/// Versioned on `parse_typed_array` in the platform version's document type
+/// schema versions. `None` selects the behavior of the versions that predate
+/// typed arrays: nothing is parsed here, so the scalar parser refuses an array
+/// that is not a byte array with the refusal it always had.
+///
+/// # Parameters
+///
+/// * `inner_properties`: the property schema as a value map
+/// * `platform_version`: selects the generation
+///
+/// # Returns
+///
+/// The typed array property type, or `None` when the property is not a typed
+/// array or the version does not parse one.
+pub(crate) fn parse_typed_array(
+    inner_properties: &BTreeMap<String, &Value>,
+    platform_version: &PlatformVersion,
+) -> Result<Option<DocumentPropertyType>, DataContractError> {
+    match platform_version
+        .dpp
+        .contract_versions
+        .document_type_versions
+        .schema
+        .parse_typed_array
+    {
+        None => Ok(None),
+        Some(0) => v0::parse_typed_array_v0(inner_properties),
+        Some(version) => Err(DataContractError::Unsupported(format!(
+            "parse_typed_array version {version} is not supported"
+        ))),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::data_contract::document_type::array::ArrayItemType;
+    use crate::data_contract::document_type::TypedArrayProperty;
+    use platform_value::platform_value;
+
+    #[test]
+    fn should_parse_a_typed_array_from_protocol_version_14_and_leave_it_alone_before() {
+        let schema = platform_value!({
+            "type": "array",
+            "minItems": 1,
+            "maxItems": 8,
+            "uniqueItems": true,
+            "items": { "type": "string", "maxLength": 16 }
+        });
+        let map = schema.to_btree_ref_string_map().unwrap();
+
+        assert_eq!(
+            parse_typed_array(&map, PlatformVersion::latest()).unwrap(),
+            Some(DocumentPropertyType::TypedArray(TypedArrayProperty {
+                items: ArrayItemType::String(None, Some(16)),
+                min_items: Some(1),
+                max_items: Some(8),
+                unique_items: true,
+            }))
+        );
+        // Protocol version 13 leaves the property to the scalar parser
+        let v13 = PlatformVersion::get(13).unwrap();
+        assert_eq!(parse_typed_array(&map, v13).unwrap(), None);
+    }
+}