feat(dpp)!: typed scalar arrays in document schemas (PV14)

book/src/data-model/documents.md

@@ -305,6 +305,33 @@ Enforcement lives in the replace action's state validation (generation 1). The a
 
 In Rust the lists are `DocumentTypeV2Getters::immutable_fields()` and `immutable_fields_allow_setting()`. Earlier document type generations return empty sets.
 
+## Typed Arrays
+
+Up to protocol version 13 a `type: "array"` property had to be a byte array (`byteArray: true`). Protocol version 14 adds typed arrays: a list whose `items` schema says what every element is.
+
+```json
+"reasons": {
+  "type": "array",
+  "minItems": 0,
+  "maxItems": 64,
+  "uniqueItems": true,
+  "items": {
+    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
+    "contentMediaType": "application/x.dash.dpp.identifier"
+  },
+  "position": 2
+}
+```
+
+- An element is a scalar: an integer, a number, a string (with `minLength` / `maxLength`), a boolean, a byte array (`byteArray: true`, whose `minItems` / `maxItems` count bytes) or an identifier. Objects and arrays of arrays are refused, and so is `refersTo` on an element for now. An element may be limited to allowed values with `enum`; `const` is refused on elements, since a one-value `enum` does the same and a contract update can still widen it.
+- On the array itself `minItems` and `maxItems` count elements, not bytes. `maxItems` is required, `minItems` may not exceed it and `contentMediaType` belongs on the items; these hold on every parse. Contract registration also caps `maxItems` at `SystemLimits::max_typed_array_items` (1024), so a typed array's worst-case size, which fee estimation charges by, stays small. `uniqueItems: true` refuses a document that repeats an element.
+- A byte array keeps its form and takes no `items`. On a plain byte array `uniqueItems` keeps its old meaning, no repeated byte, but an identifier (a byte array with the identifier `contentMediaType`) refuses it: an identifier is one value, and "no repeated byte" would refuse most of them.
+- The document is validated against the JSON schema as always, so a list that is too long, too short, repeats an element under `uniqueItems` or holds a wrong-typed element fails with the usual `JsonSchemaError`.
+
+The array is stored inline in the document, like any other property: a varint element count followed by each element in its own encoding (see [Document Serialization](../serialization/document-serialization.md)). Identifier and byte array elements are conversion paths (`reasons[]`, `find_identifier_and_binary_paths` 1), so a document built from JSON or a value map converts every element, as it converts a scalar identifier. Nothing is written per element, so a typed array cannot be an index property (`InvalidIndexPropertyTypeError`), an indexOnly terminal or entry payload property, or one side of a `propertyAgreement`.
+
+In Rust a typed array parses to `DocumentPropertyType::TypedArray(TypedArrayProperty)`, with the element type as an `ArrayItemType`. The parse is the versioned `parse_typed_array` (`None` before protocol version 14, where an array that is not a byte array is refused as it always was). The older `DocumentPropertyType::Array` variant has the same encoding without the count bounds, and the parser never produces it.
+
 ## Rules and Guidelines
 
 **Do:**

book/src/serialization/document-serialization.md

@@ -184,7 +184,7 @@ All numeric values use **big-endian** byte order.
 | `byteArray` (variable size) | varint length prefix + raw bytes |
 | `identifier` | 32 bytes raw |
 | `date` | 8 bytes big-endian f64 (when optional: `0xff` prefix + 8 bytes) |
-| `array` | varint element count + each element encoded in sequence |
+| `array` (typed array, protocol v14) | varint element count + each element in sequence: integer or number 8 bytes, boolean 1 byte, string, byte array or identifier a varint length prefix + the bytes (an identifier element is always 33 bytes) |
 | `object` | Nested fields serialized recursively in their schema position order |
 
 **Note on date types**: User-property `date` fields are encoded as **f64** (8 bytes). System timestamps (`$createdAt`, `$updatedAt`, `$transferredAt`) are **u64** milliseconds. Both are 8 bytes big-endian but use different numeric representations.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://github.com/dashpay/platform/blob/master/packages/rs-dpp/schema/meta_schemas/document/v1/document-meta.json",
-  "$comment": "EDITABLE UNTIL 4.2 (PROTOCOL V14) IS LIVE ON MAINNET; FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties, the requiredSince property keyword (the contract version a property is required from), and the timeRange index transform, and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once 4.2 is live on mainnet, mutating it would change historical validation results and break consensus replay, and any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
+  "$comment": "EDITABLE UNTIL 4.2 (PROTOCOL V14) IS LIVE ON MAINNET; FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties, the requiredSince property keyword (the contract version a property is required from), the timeRange index transform, and typed arrays (an array property whose items schema names one scalar element type instead of byteArray, stored inline as an element count followed by the elements), and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once 4.2 is live on mainnet, mutating it would change historical validation results and break consensus replay, and any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
   "type": "object",
   "$defs": {
     "documentProperties": {
@@ -91,6 +91,10 @@
         "uniqueItems": {
           "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/uniqueItems"
         },
+        "items": {
+          "description": "typed arrays only: the schema every element of the array has",
+          "$ref": "#/$defs/documentArrayItem"
+        },
         "refersTo": {
           "type": "object",
           "properties": {
@@ -327,6 +331,18 @@
             }
           }
         },
+        "items": {
+          "description": "should be used only with array type",
+          "properties": {
+            "type": {
+              "type": "string",
+              "const": "array"
+            }
+          },
+          "required": [
+            "type"
+          ]
+        },
         "contentMediaType": {
           "if": {
             "properties": {
@@ -336,6 +352,7 @@
             }
           },
           "then": {
+            "$comment": "an identifier is one value, not a list of bytes: uniqueItems would refuse every identifier that repeats a byte, most of them",
             "properties": {
               "byteArray": {
                 "const": true
@@ -345,7 +362,8 @@
               },
               "maxItems": {
                 "const": 32
-              }
+              },
+              "uniqueItems": false
             },
             "required": [
               "byteArray",
@@ -439,7 +457,7 @@
           }
         },
         {
-          "$comment": "allow only byte arrays",
+          "$comment": "an array is a byte array or a typed array. A byte array declares byteArray: true, its minItems and maxItems count bytes, and it takes no items (an identifier byte array takes no uniqueItems either, see contentMediaType). A typed array declares items, the schema of every element, instead; its minItems and maxItems count elements, and maxItems is required",
           "if": {
             "properties": {
               "type": {
@@ -451,12 +469,25 @@
             ]
           },
           "then": {
-            "properties": {
-              "byteArray": true
+            "if": {
+              "required": [
+                "byteArray"
+              ]
             },
-            "required": [
-              "byteArray"
-            ]
+            "then": {
+              "properties": {
+                "items": false
+              }
+            },
+            "else": {
+              "properties": {
+                "contentMediaType": false
+              },
+              "required": [
+                "items",
+                "maxItems"
+              ]
+            }
           }
         },
         {
@@ -493,6 +524,160 @@
         }
       ]
     },
+    "documentArrayItem": {
+      "$comment": "The element schema of a typed array: one scalar, an integer, a number, a string, a boolean, or a byte array (byteArray: true), which with the identifier contentMediaType is an identifier. Objects and arrays of arrays are refused. An element carries no position, requiredSince, refersTo, uniqueItems, const or examples of its own (a one-value enum does what const would, and an update can widen it; examples annotate nothing an element needs)",
+      "type": "object",
+      "properties": {
+        "$comment": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/core#/properties/$comment"
+        },
+        "description": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/meta-data#/properties/description"
+        },
+        "type": {
+          "enum": [
+            "integer",
+            "number",
+            "string",
+            "boolean",
+            "array"
+          ]
+        },
+        "multipleOf": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/multipleOf"
+        },
+        "maximum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/maximum"
+        },
+        "exclusiveMaximum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/exclusiveMaximum"
+        },
+        "minimum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/minimum"
+        },
+        "exclusiveMinimum": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/exclusiveMinimum"
+        },
+        "maxLength": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/maxLength"
+        },
+        "minLength": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/minLength"
+        },
+        "pattern": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/pattern"
+        },
+        "maxItems": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/maxItems"
+        },
+        "minItems": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/validation#/properties/minItems"
+        },
+        "enum": {
+          "type": "array",
+          "items": true,
+          "minItems": 1,
+          "uniqueItems": true
+        },
+        "format": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/format-annotation#/properties/format"
+        },
+        "contentMediaType": {
+          "$ref": "https://json-schema.org/draft/2020-12/meta/content#/properties/contentMediaType"
+        },
+        "byteArray": {
+          "type": "boolean",
+          "const": true
+        }
+      },
+      "required": [
+        "type"
+      ],
+      "additionalProperties": false,
+      "dependentSchemas": {
+        "byteArray": {
+          "description": "should be used only with array type",
+          "properties": {
+            "type": {
+              "const": "array"
+            }
+          }
+        },
+        "contentMediaType": {
+          "if": {
+            "properties": {
+              "contentMediaType": {
+                "const": "application/x.dash.dpp.identifier"
+              }
+            }
+          },
+          "then": {
+            "properties": {
+              "byteArray": {
+                "const": true
+              },
+              "minItems": {
+                "const": 32
+              },
+              "maxItems": {
+                "const": 32
+              }
+            },
+            "required": [
+              "byteArray",
+              "minItems",
+              "maxItems"
+            ]
+          }
+        },
+        "pattern": {
+          "description": "prevent slow pattern matching of large strings",
+          "properties": {
+            "maxLength": {
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 50000
+            }
+          },
+          "required": [
+            "maxLength"
+          ]
+        },
+        "format": {
+          "description": "prevent slow format validation of large strings",
+          "properties": {
+            "maxLength": {
+              "type": "integer",
+              "minimum": 0,
+              "maximum": 50000
+            }
+          },
+          "required": [
+            "maxLength"
+          ]
+        }
+      },
+      "allOf": [
+        {
+          "$comment": "an array element is a byte array: a typed array cannot hold another typed array",
+          "if": {
+            "properties": {
+              "type": {
+                "const": "array"
+              }
+            },
+            "required": [
+              "type"
+            ]
+          },
+          "then": {
+            "required": [
+              "byteArray"
+            ]
+          }
+        }
+      ]
+    },
     "documentActionTokenCost": {
       "type": "object",
       "properties": {

packages/rs-dpp/src/data_contract/document_type/class_methods/mod.rs

@@ -8,6 +8,7 @@ use crate::ProtocolError;
 
 pub(crate) mod apply_required_since;
 mod create_document_types_from_document_schemas;
+mod parse_typed_array;
 mod should_use_creator_id;
 mod system_properties;
 mod try_from_schema;

packages/rs-dpp/src/data_contract/document_type/class_methods/parse_typed_array/mod.rs

@@ -0,0 +1,75 @@
+use std::collections::BTreeMap;
+
+use platform_value::Value;
+use platform_version::version::PlatformVersion;
+
+use crate::data_contract::document_type::DocumentPropertyType;
+use crate::data_contract::errors::DataContractError;
+
+mod v0;
+
+/// Parses a typed array property: `type: "array"` with an `items` element
+/// schema in place of `byteArray`, into [`DocumentPropertyType::TypedArray`].
+///
+/// Returns `None` for every other property, a byte array included, which the
+/// caller leaves to `DocumentPropertyType::try_from_value_map`.
+///
+/// Versioned on `parse_typed_array` in the platform version's document type
+/// schema versions. `None` selects the behavior of the versions that predate
+/// typed arrays: nothing is parsed here, so `try_from_value_map` refuses an
+/// array that is not a byte array, exactly as those versions always did.
+pub(crate) fn parse_typed_array(
+    inner_properties: &BTreeMap<String, &Value>,
+    platform_version: &PlatformVersion,
+) -> Result<Option<DocumentPropertyType>, DataContractError> {
+    match platform_version
+        .dpp
+        .contract_versions
+        .document_type_versions
+        .schema
+        .parse_typed_array
+    {
+        None => Ok(None),
+        Some(0) => v0::parse_typed_array_v0(inner_properties),
+        Some(version) => Err(DataContractError::Unsupported(format!(
+            "parse_typed_array version {version} is not supported"
+        ))),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::data_contract::document_type::array::{ArrayItemType, TypedArrayProperty};
+    use platform_value::platform_value;
+
+    #[test]
+    fn should_parse_a_typed_array_from_protocol_version_14_and_leave_it_alone_before() {
+        let schema = platform_value!({
+            "type": "array",
+            "minItems": 1,
+            "maxItems": 8,
+            "uniqueItems": true,
+            "items": { "type": "string", "maxLength": 16 }
+        });
+        let map = schema
+            .to_btree_ref_string_map()
+            .expect("the schema is a map");
+
+        assert_eq!(
+            parse_typed_array(&map, PlatformVersion::latest()).expect("parses"),
+            Some(DocumentPropertyType::TypedArray(TypedArrayProperty {
+                item_type: ArrayItemType::String(None, Some(16)),
+                min_items: Some(1),
+                max_items: 8,
+                unique_items: true,
+            }))
+        );
+        // Protocol version 13 leaves the property to the scalar parser
+        let platform_version_13 = PlatformVersion::get(13).expect("protocol version 13 exists");
+        assert_eq!(
+            parse_typed_array(&map, platform_version_13).expect("parses"),
+            None
+        );
+    }
+}