feat(dpp)!: encode typed array elements as their scalar property type (PV14)

book/src/data-model/documents.md

@@ -328,9 +328,9 @@ Up to protocol version 13 a `type: "array"` property had to be a byte array (`by
 - A byte array keeps its form and takes no `items`. On a plain byte array `uniqueItems` keeps its old meaning, no repeated byte, but an identifier (a byte array with the identifier `contentMediaType`) refuses it: an identifier is one value, and "no repeated byte" would refuse most of them.
 - The document is validated against the JSON schema as always, so a list that is too long, too short, repeats an element under `uniqueItems` or holds a wrong-typed element fails with the usual `JsonSchemaError`.
 
-The array is stored inline in the document, like any other property: a varint element count followed by each element in its own encoding (see [Document Serialization](../serialization/document-serialization.md)). Identifier and byte array elements are conversion paths (`reasons[]`, `find_identifier_and_binary_paths` 1), so a document built from JSON or a value map converts every element, as it converts a scalar identifier. Nothing is written per element, so a typed array cannot be an index property (`InvalidIndexPropertyTypeError`), an indexOnly terminal or entry payload property, or one side of a `propertyAgreement`.
+The array is stored inline in the document, like any other property: a varint element count followed by the elements, each encoded exactly as a required property of the element's type (see [Document Serialization](../serialization/document-serialization.md)). The `reasons` list above is therefore one count byte and 32 raw bytes per identifier, and an integer element bounded `0`..`100` takes one byte. Since the stored bytes depend on the element's type, a contract update may not change how an element encodes: raising an integer element's `maximum` (or adding an `enum` value) past its width, or unpinning a fixed-size byte array element, is refused with `DocumentTypeUpdateError`. A longer `maxLength`, a larger `maxItems` or a raised `maximum` that keeps the width are accepted. Identifier and byte array elements are conversion paths (`reasons[]`, `find_identifier_and_binary_paths` 1), so a document built from JSON or a value map converts every element, as it converts a scalar identifier. Nothing is written per element, so a typed array cannot be an index property (`InvalidIndexPropertyTypeError`), an indexOnly terminal or entry payload property, or one side of a `propertyAgreement`.
 
-In Rust a typed array parses to `DocumentPropertyType::TypedArray(TypedArrayProperty)`, with the element type as an `ArrayItemType`. The parse is the versioned `parse_typed_array` (`None` before protocol version 14, where an array that is not a byte array is refused as it always was). The older `DocumentPropertyType::Array` variant has the same encoding without the count bounds, and the parser never produces it.
+In Rust a typed array parses to `DocumentPropertyType::TypedArray(TypedArrayProperty)`, whose `item_type` is the `DocumentPropertyType` the `items` schema parses to as a property schema (`try_from_value_map` with the contract's parsing options). The parse is the versioned `parse_typed_array` (`None` before protocol version 14, where an array that is not a byte array is refused as it always was). The older `DocumentPropertyType::Array` variant, whose elements are an `ArrayItemType` in their own length-prefixed encoding, is never produced by the parser.
 
 ## Rules and Guidelines
 

book/src/serialization/document-serialization.md

@@ -184,7 +184,7 @@ All numeric values use **big-endian** byte order.
 | `byteArray` (variable size) | varint length prefix + raw bytes |
 | `identifier` | 32 bytes raw |
 | `date` | 8 bytes big-endian f64 (when optional: `0xff` prefix + 8 bytes) |
-| `array` (typed array, protocol v14) | varint element count + each element in sequence: integer or number 8 bytes, boolean 1 byte, string, byte array or identifier a varint length prefix + the bytes (an identifier element is always 33 bytes) |
+| `array` (typed array, protocol v14) | varint element count + each element encoded exactly as a required property of the element's type (rows above): an identifier element is 32 raw bytes, an integer element takes the width its bounds give it, a fixed-size byte array element is raw, a string or variable-size byte array element has a varint length prefix. Elements never carry a presence byte |
 | `object` | Nested fields serialized recursively in their schema position order |
 
 **Note on date types**: User-property `date` fields are encoded as **f64** (8 bytes). System timestamps (`$createdAt`, `$updatedAt`, `$transferredAt`) are **u64** milliseconds. Both are 8 bytes big-endian but use different numeric representations.
@@ -284,4 +284,6 @@ See `packages/rs-scripts/README.md` for full usage details.
 
 6. **ByteArray encoding depends on size constraints.** Fixed-size byte arrays (where `minItems == maxItems` in the schema) have no length prefix. Variable-size byte arrays have a varint length prefix. Check the schema to know which encoding is used.
 
-7. **In version 3, the same document type can produce different property layouts.** A property annotated with `requiredSince` is presence-flagged in documents stamped below the annotation and raw in documents stamped at or above it. Two version-3 documents of the same type may therefore differ in layout — always read the stamp varint and resolve each property's requiredness against it before decoding the properties section.
+7. **A typed array's element width comes from its `items` schema.** Each element is laid out as a required property of the element's type, so an integer element bounded `0`..`100` is 1 byte and an unbounded one 8, and a fixed-size byte array or identifier element has no length prefix. Parse the `items` schema exactly as a property schema to know the width, including the contract's `sizedIntegerTypes` setting.
+
+8. **In version 3, the same document type can produce different property layouts.** A property annotated with `requiredSince` is presence-flagged in documents stamped below the annotation and raw in documents stamped at or above it. Two version-3 documents of the same type may therefore differ in layout — always read the stamp varint and resolve each property's requiredness against it before decoding the properties section.

packages/rs-dpp/src/data_contract/document_type/class_methods/parse_typed_array/mod.rs

@@ -3,7 +3,9 @@ use std::collections::BTreeMap;
 use platform_value::Value;
 use platform_version::version::PlatformVersion;
 
-use crate::data_contract::document_type::DocumentPropertyType;
+use crate::data_contract::document_type::{
+    DocumentPropertyType, DocumentPropertyTypeParsingOptions,
+};
 use crate::data_contract::errors::DataContractError;
 
 mod v0;
@@ -12,14 +14,17 @@ mod v0;
 /// schema in place of `byteArray`, into [`DocumentPropertyType::TypedArray`].
 ///
 /// Returns `None` for every other property, a byte array included, which the
-/// caller leaves to `DocumentPropertyType::try_from_value_map`.
+/// caller leaves to `DocumentPropertyType::try_from_value_map`. The element
+/// schema is parsed by that same scalar parser with the property's `options`,
+/// so an element has the type a scalar property of its schema would have.
 ///
 /// Versioned on `parse_typed_array` in the platform version's document type
 /// schema versions. `None` selects the behavior of the versions that predate
 /// typed arrays: nothing is parsed here, so `try_from_value_map` refuses an
 /// array that is not a byte array, exactly as those versions always did.
 pub(crate) fn parse_typed_array(
     inner_properties: &BTreeMap<String, &Value>,
+    options: &DocumentPropertyTypeParsingOptions,
     platform_version: &PlatformVersion,
 ) -> Result<Option<DocumentPropertyType>, DataContractError> {
     match platform_version
@@ -30,7 +35,7 @@ pub(crate) fn parse_typed_array(
         .parse_typed_array
     {
         None => Ok(None),
-        Some(0) => v0::parse_typed_array_v0(inner_properties),
+        Some(0) => v0::parse_typed_array_v0(inner_properties, options),
         Some(version) => Err(DataContractError::Unsupported(format!(
             "parse_typed_array version {version} is not supported"
         ))),
@@ -40,7 +45,8 @@ pub(crate) fn parse_typed_array(
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::data_contract::document_type::array::{ArrayItemType, TypedArrayProperty};
+    use crate::data_contract::document_type::array::TypedArrayProperty;
+    use crate::data_contract::document_type::StringPropertySizes;
     use platform_value::platform_value;
 
     #[test]
@@ -55,11 +61,15 @@ mod tests {
         let map = schema
             .to_btree_ref_string_map()
             .expect("the schema is a map");
+        let options = DocumentPropertyTypeParsingOptions::default();
 
         assert_eq!(
-            parse_typed_array(&map, PlatformVersion::latest()).expect("parses"),
+            parse_typed_array(&map, &options, PlatformVersion::latest()).expect("parses"),
             Some(DocumentPropertyType::TypedArray(TypedArrayProperty {
-                item_type: ArrayItemType::String(None, Some(16)),
+                item_type: Box::new(DocumentPropertyType::String(StringPropertySizes {
+                    min_length: None,
+                    max_length: Some(16),
+                })),
                 min_items: Some(1),
                 max_items: 8,
                 unique_items: true,
@@ -68,7 +78,7 @@ mod tests {
         // Protocol version 13 leaves the property to the scalar parser
         let platform_version_13 = PlatformVersion::get(13).expect("protocol version 13 exists");
         assert_eq!(
-            parse_typed_array(&map, platform_version_13).expect("parses"),
+            parse_typed_array(&map, &options, platform_version_13).expect("parses"),
             None
         );
     }

packages/rs-dpp/src/data_contract/document_type/class_methods/parse_typed_array/v0/mod.rs

@@ -3,8 +3,10 @@ use std::collections::BTreeMap;
 use platform_value::btreemap_extensions::BTreeValueMapHelper;
 use platform_value::Value;
 
-use crate::data_contract::document_type::array::{ArrayItemType, TypedArrayProperty};
-use crate::data_contract::document_type::{property_names, DocumentPropertyType};
+use crate::data_contract::document_type::array::TypedArrayProperty;
+use crate::data_contract::document_type::{
+    property_names, DocumentPropertyType, DocumentPropertyTypeParsingOptions,
+};
 use crate::data_contract::errors::DataContractError;
 
 /// Generation 0 parse rules: an array property that does not declare
@@ -17,6 +19,7 @@ use crate::data_contract::errors::DataContractError;
 /// validation.
 pub(super) fn parse_typed_array_v0(
     inner_properties: &BTreeMap<String, &Value>,
+    options: &DocumentPropertyTypeParsingOptions,
 ) -> Result<Option<DocumentPropertyType>, DataContractError> {
     let is_array = inner_properties
         .get(property_names::TYPE)
@@ -43,7 +46,7 @@ pub(super) fn parse_typed_array_v0(
         ));
     }
 
-    let item_type = ArrayItemType::try_from(*items)?;
+    let item_type = parse_element_type(items, options)?;
 
     // Fee estimation sizes the inline list by its bound
     let Some(max_items) = inner_properties.get_optional_integer(property_names::MAX_ITEMS)? else {
@@ -61,7 +64,7 @@ pub(super) fn parse_typed_array_v0(
     }
 
     Ok(Some(DocumentPropertyType::TypedArray(TypedArrayProperty {
-        item_type,
+        item_type: Box::new(item_type),
         min_items,
         max_items,
         unique_items: inner_properties
@@ -70,6 +73,80 @@ pub(super) fn parse_typed_array_v0(
     })))
 }
 
+/// The element type of a typed array: its `items` schema parsed exactly as a
+/// scalar property schema is, so an integer element takes the width its
+/// bounds give it and a byte array element with the identifier media type is
+/// an identifier. Objects and arrays of arrays are refused.
+///
+/// `refersTo` is refused for now. A reference on identifier elements would be
+/// read from this same map and folded into the element type, as
+/// `apply_property_reference` folds one into a scalar identifier.
+fn parse_element_type(
+    items: &Value,
+    options: &DocumentPropertyTypeParsingOptions,
+) -> Result<DocumentPropertyType, DataContractError> {
+    // The tuple form (`items: [..]`) and boolean schemas are not one element
+    // schema
+    let items_map = items.to_btree_ref_string_map().map_err(|_| {
+        DataContractError::InvalidContractStructure(
+            "the items of a typed array must be one element schema (an object)".to_string(),
+        )
+    })?;
+    if items_map.contains_key(property_names::REF) {
+        return Err(DataContractError::InvalidContractStructure(
+            "the items of a typed array must be an inline element schema, not a $ref".to_string(),
+        ));
+    }
+    if items_map.contains_key(property_names::REFERS_TO) {
+        return Err(DataContractError::InvalidContractStructure(
+            "refersTo is not supported on the elements of a typed array".to_string(),
+        ));
+    }
+    match items_map
+        .get(property_names::TYPE)
+        .and_then(|type_value| type_value.as_text())
+    {
+        Some("object") => {
+            return Err(DataContractError::InvalidContractStructure(
+                "arrays of objects are not supported: the elements of a typed array must be \
+                 scalars (integer, number, string, boolean, byte array or identifier)"
+                    .to_string(),
+            ))
+        }
+        Some("array") if !items_map.contains_key(property_names::BYTE_ARRAY) => {
+            return Err(DataContractError::InvalidContractStructure(
+                "arrays of arrays are not supported: an element of a typed array may be a byte \
+                 array (byteArray: true) or an identifier, but not another array"
+                    .to_string(),
+            ))
+        }
+        _ => {}
+    }
+
+    let element_type = DocumentPropertyType::try_from_value_map(&items_map, options)?;
+    match element_type {
+        DocumentPropertyType::U128
+        | DocumentPropertyType::I128
+        | DocumentPropertyType::U64
+        | DocumentPropertyType::I64
+        | DocumentPropertyType::U32
+        | DocumentPropertyType::I32
+        | DocumentPropertyType::U16
+        | DocumentPropertyType::I16
+        | DocumentPropertyType::U8
+        | DocumentPropertyType::I8
+        | DocumentPropertyType::F64
+        | DocumentPropertyType::String(_)
+        | DocumentPropertyType::ByteArray(_)
+        | DocumentPropertyType::Identifier
+        | DocumentPropertyType::Boolean => Ok(element_type),
+        other => Err(DataContractError::InvalidContractStructure(format!(
+            "unsupported typed array element type: {}",
+            other.name()
+        ))),
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -79,7 +156,7 @@ mod tests {
         let map = schema
             .to_btree_ref_string_map()
             .expect("the schema is a map");
-        parse_typed_array_v0(&map)
+        parse_typed_array_v0(&map, &DocumentPropertyTypeParsingOptions::default())
     }
 
     #[test]
@@ -105,14 +182,68 @@ mod tests {
             }))
             .expect("parses"),
             Some(DocumentPropertyType::TypedArray(TypedArrayProperty {
-                item_type: ArrayItemType::Integer,
+                item_type: Box::new(DocumentPropertyType::I64),
                 min_items: Some(1),
                 max_items: 4,
                 unique_items: false,
             }))
         );
     }
 
+    /// An element is parsed by the scalar parser, so it takes the type a
+    /// scalar property of the same schema takes: an integer sized by its
+    /// bounds (when the contract sizes integers), an identifier from the
+    /// identifier media type.
+    #[test]
+    fn should_type_an_element_as_a_scalar_property_of_its_schema() {
+        for (items, sized_integer_types, expected) in [
+            (
+                platform_value!({ "type": "integer", "minimum": 0, "maximum": 100 }),
+                true,
+                DocumentPropertyType::U8,
+            ),
+            (
+                platform_value!({ "type": "integer", "minimum": -1000, "maximum": 1000 }),
+                true,
+                DocumentPropertyType::I16,
+            ),
+            // A contract that does not size integers keeps them at 64 bits
+            (
+                platform_value!({ "type": "integer", "minimum": 0, "maximum": 100 }),
+                false,
+                DocumentPropertyType::I64,
+            ),
+            (
+                platform_value!({
+                    "type": "array",
+                    "byteArray": true,
+                    "minItems": 32,
+                    "maxItems": 32,
+                    "contentMediaType": "application/x.dash.dpp.identifier"
+                }),
+                true,
+                DocumentPropertyType::Identifier,
+            ),
+        ] {
+            let schema =
+                platform_value!({ "type": "array", "maxItems": 4, "items": items.clone() });
+            let map = schema
+                .to_btree_ref_string_map()
+                .expect("the schema is a map");
+            let parsed = parse_typed_array_v0(
+                &map,
+                &DocumentPropertyTypeParsingOptions {
+                    sized_integer_types,
+                },
+            )
+            .expect("parses");
+            let Some(DocumentPropertyType::TypedArray(typed_array)) = parsed else {
+                panic!("{items:?} should parse to a typed array");
+            };
+            assert_eq!(*typed_array.item_type, expected, "{items:?}");
+        }
+    }
+
     #[test]
     fn should_refuse_a_typed_array_missing_items_or_max_items_or_with_a_misplaced_bound() {
         for (schema, fragment) in [

packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/mod.rs

@@ -7,7 +7,7 @@ use crate::data_contract::document_type::{
     is_referenced_system_agreement_property, is_referring_system_agreement_property,
     property_names, ContractReferenceModeration, ContractReferenceOwner,
     ContractReferenceRequirements, DocumentProperty, DocumentPropertyReferenceTarget,
-    DocumentPropertyType, DocumentType,
+    DocumentPropertyType, DocumentPropertyTypeParsingOptions, DocumentType,
 };
 use crate::data_contract::errors::DataContractError;
 use crate::data_contract::{TokenConfiguration, TokenContractPosition};
@@ -155,9 +155,11 @@ fn insert_values(
             platform_version,
         )?;
 
-        let property_type = match parse_typed_array(&inner_properties, platform_version)? {
+        let options: DocumentPropertyTypeParsingOptions = config.into();
+        let property_type = match parse_typed_array(&inner_properties, &options, platform_version)?
+        {
             Some(typed_array) => typed_array,
-            None => DocumentPropertyType::try_from_value_map(&inner_properties, &config.into())?,
+            None => DocumentPropertyType::try_from_value_map(&inner_properties, &options)?,
         };
 
         match property_type {
@@ -237,9 +239,10 @@ fn insert_values_nested(
         platform_version,
     )?;
 
-    let property_type = match parse_typed_array(&inner_properties, platform_version)? {
+    let options: DocumentPropertyTypeParsingOptions = config.into();
+    let property_type = match parse_typed_array(&inner_properties, &options, platform_version)? {
         Some(typed_array) => typed_array,
-        None => DocumentPropertyType::try_from_value_map(&inner_properties, &config.into())?,
+        None => DocumentPropertyType::try_from_value_map(&inner_properties, &options)?,
     };
 
     let property_type = match property_type {