feat(platform)!: refersTo on typed array elements (PV14)

book/src/data-model/documents.md

@@ -342,7 +342,7 @@ Up to protocol version 13 a `type: "array"` property had to be a byte array (`by
 }
 ```
 
-- An element is a scalar: an integer, a number, a string (with `minLength` / `maxLength`), a boolean, a byte array (`byteArray: true`, whose `minItems` / `maxItems` count bytes) or an identifier. Objects and arrays of arrays are refused, and so is `refersTo` on an element for now. An element may be limited to allowed values with `enum`; `const` is refused on elements, since a one-value `enum` does the same and a contract update can still widen it. The parser reads an element's `enum`, `minimum` and `maximum` onto the typed array (`ArrayItemConstraints`), refusing on both parse paths an `enum` with no member or a member of another type, an `enum` on a byte array or identifier element, and a `minimum` above the `maximum`, so random document generation stays inside them; the JSON schema validator enforces them on every document.
+- An element is a scalar: an integer, a number, a string (with `minLength` / `maxLength`), a boolean, a byte array (`byteArray: true`, whose `minItems` / `maxItems` count bytes) or an identifier. Objects and arrays of arrays are refused. An identifier element may carry `refersTo` (see [References on the Elements](#references-on-the-elements)). An element may be limited to allowed values with `enum`; `const` is refused on elements, since a one-value `enum` does the same and a contract update can still widen it. The parser reads an element's `enum`, `minimum` and `maximum` onto the typed array (`ArrayItemConstraints`), refusing on both parse paths an `enum` with no member or a member of another type, an `enum` on a byte array or identifier element, and a `minimum` above the `maximum`, so random document generation stays inside them; the JSON schema validator enforces them on every document.
 - On the array itself `minItems` and `maxItems` count elements, not bytes. `maxItems` is required, `minItems` may not exceed it and `contentMediaType` belongs on the items; these hold on every parse. Contract registration also caps `maxItems` at `SystemLimits::max_typed_array_items` (1024), so a typed array's worst-case size, which fee estimation charges by, stays small. `uniqueItems: true` refuses a document that repeats an element.
 - A byte array keeps its form and takes no `items`. On a plain byte array `uniqueItems` keeps its old meaning, no repeated byte, but an identifier (a byte array with the identifier `contentMediaType`) refuses it: an identifier is one value, and "no repeated byte" would refuse most of them.
 - The document is validated against the JSON schema as always, so a list that is too long, too short, repeats an element under `uniqueItems` or holds a wrong-typed element fails with the usual `JsonSchemaError`.
@@ -351,6 +351,27 @@ The array is stored inline in the document, like any other property: a varint el
 
 In Rust a typed array parses to `DocumentPropertyType::TypedArray(TypedArrayProperty)`, whose `item_type` is the `DocumentPropertyType` the `items` schema parses to as a property schema (`try_from_value_map` with the contract's parsing options). The parse is the versioned `parse_typed_array` (`None` before protocol version 14, where an array that is not a byte array is refused as it always was). The older `DocumentPropertyType::Array` variant, whose elements are an `ArrayItemType` in their own length-prefixed encoding, is never produced by the parser.
 
+### References on the Elements
+
+An identifier element may carry a `refersTo` declaration, which every element of the list then declares. The moderation charters' `reasons` refers to `reason` documents this way:
+
+```json
+"items": {
+  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
+  "contentMediaType": "application/x.dash.dpp.identifier",
+  "refersTo": { "type": "permanentDocument", "documentType": "reason" }
+}
+```
+
+- The declaration takes every target and key a single identifier property's `refersTo` takes (`identity`, `contract` with `contractRequirements`, `token`, and `permanentDocument` and `deletableDocument` with `contractId`, `documentType` and `propertyAgreement`), with the same checks at contract registration, except `identityPublicKey` in either form: its `keyIdProperty` names one sibling key id, and the `identityProperty` form sits on the key id itself, neither of which can pair with many elements, so an element may not declare it. The declaration belongs on the `items`; on the array itself it is refused.
+- When a document is created or replaced each element is checked as a single reference is, in list order: the target must exist, a referenced contract must meet the `contractRequirements`, a referenced document's type must be deletable or not as declared, and each `propertyAgreement` pair must hold. The referring side of a pair is still a property of the referring document or its `$ownerId`, the same for every element; the referenced side is a property of that element's referenced document. The first element that fails refuses the write with the error a single reference gives (`ReferencedEntityNotFoundError` 40120, `ReferencedDocumentPropertyMismatchError` 40127, `ReferencedContractRequirementNotMetError` 40135 and so on), whose `path` names the element by its list path: `reasons[2]` for the third. An empty or absent list checks nothing. Registration errors name the declaration `submittedCharter.reasons[]`.
+- A replace follows the rules of a single reference, element by element. A changed list re-validates the elements the stored list did not hold (the replace action carries the stored value of each changed property, `stored_changed_values`); the ones it held are unchanged references and are left alone, as an unchanged single reference is. Every element is re-validated when a property bound by a `propertyAgreement` changed, and on every replace when an agreement is keyed by `$ownerId` or the elements are `deletableDocument` references. An element repeating an earlier one of the same list is not fetched again.
+- Every read is billed as a single reference's is. A foreign contract holding the referenced document type is fetched once per list, not once per element. Contract registration caps the references one document of a type can carry at `SystemLimits::max_references_per_document` (256), counting one per property with `refersTo` (an identifier, or a key id carrying a key reference) and `maxItems` per typed array of referencing elements: `maxItems` alone would let a type declare many lists of up to 1024 references each, and each one is a read when a document is written.
+- An `immutable` property may not hold a `deletableDocument` reference a replace could not clear: a typed array of them, at the top level or inside an immutable object, or a single one inside an immutable object. Every replace re-validates them, so once a target is deleted the property would have to change, which an immutable property cannot. A single `deletableDocument` reference that is itself the immutable top-level property has a way out, a replace may remove it once its target is gone, and that exception reads the one identifier the removed top-level property held.
+- A changed element `refersTo` is an incompatible schema change on contract update, as a changed `refersTo` on a scalar is.
+
+In Rust the element parses to `DocumentPropertyType::IdentifierWithReference(target)` inside `item_type`, through the same versioned `apply_property_reference` a scalar identifier goes through. `DocumentPropertyType::reference()` reports a property's declaration as `PropertyReference::Value(target)` for a scalar and `PropertyReference::Elements(target)` for a typed array; the registration check (`validate_data_contract_references`) and the write-time check (`validate_document_references`) both enumerate references through it.
+
 ## Distinct Identifier Properties
 
 Protocol version 14 adds the property-level `distinctFrom` keyword, a pure structure rule on identifier properties: the property's value must differ from the value of a named property of the same document, or from the document's `$ownerId`. It sits next to the reference keywords (`refersTo` and its `propertyAgreement`, which bind a property to another document's values) but reads nothing beyond the transition being written.

packages/js-evo-sdk/README.md

@@ -207,6 +207,8 @@ for (const ref of contract.documentTypeReferences('note')) {
 contract.documentReferences;
 ```
 
+A typed array of identifiers may declare `refersTo` on its `items`, which every element then carries. Such a declaration is listed at the list path of its elements, `path: 'reasons[]'`, which is not a property path: read the list at `reasons` and treat each element as a reference. The same declaration is on the typed array's item, `contract.documentTypeTypedArrays('charter')[0].items.refersTo`. Consensus checks every element when the document is written, and a rejection names the failing element by its index, as in `reasons[2]` for the third.
+
 A document reference comes in two strengths. `permanentDocument` requires the referenced document type to declare `canBeDeleted: false`, so a reference that was accepted keeps resolving. `deletableDocument` takes the same declaration (`contractId`, `documentType`, `propertyAgreement`) and is its disjoint counterpart: the referenced type must allow deletion (`ReferencedDocumentTypeNotDeletable`, 40131, otherwise). The referenced document must exist, and the agreement must hold, when the referring document is written, but it may be deleted afterwards. Nothing blocks that deletion and nothing cleans up after it, so a reader must expect such a reference to resolve to nothing. It can never start resolving to different content: a document id commits to the nonce of its create transition, so a deleted id can not be created again. A writer may not leave it that way: every replace of the referring document re-validates the reference, touched or not, so once the target is gone the replace has to repoint it at a document that exists or clear it (`ReferencedEntityNotFound` otherwise). A writer gate is then checked against the new target, never against a missing one. On an `immutable` property clearing is the only move, and the immutable check lets that one change through. The referring document can always be deleted. A property cannot switch between the two on a contract update, and `preallocated` indexes are only available through `permanentDocument`.
 
 Declarations are only parsed from protocol version 14 onward; a contract deserialized against an earlier version reports none even when its raw schema carries the keyword.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://github.com/dashpay/platform/blob/master/packages/rs-dpp/schema/meta_schemas/document/v1/document-meta.json",
-  "$comment": "EDITABLE UNTIL 4.2 (PROTOCOL V14) IS LIVE ON MAINNET; FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties (and, for an identityPublicKey reference with identityProperty, on the key id integer property), the requiredSince property keyword (the contract version a property is required from), the timeRange index transform, and typed arrays (an array property whose items schema names one scalar element type instead of byteArray, stored inline as an element count followed by the elements), refuses `-` in property and document type names (word characters only; a census of every contract on mainnet and testnet found none), and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once 4.2 is live on mainnet, mutating it would change historical validation results and break consensus replay, and any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
+  "$comment": "EDITABLE UNTIL 4.2 (PROTOCOL V14) IS LIVE ON MAINNET; FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties (and, for an identityPublicKey reference with identityProperty, on the key id integer property), the requiredSince property keyword (the contract version a property is required from), the timeRange index transform, and typed arrays (an array property whose items schema names one scalar element type instead of byteArray, stored inline as an element count followed by the elements, whose identifier elements may carry a refersTo), refuses `-` in property and document type names (word characters only; a census of every contract on mainnet and testnet found none), and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once 4.2 is live on mainnet, mutating it would change historical validation results and break consensus replay, and any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
   "type": "object",
   "$defs": {
     "documentProperties": {
@@ -691,7 +691,7 @@
       ]
     },
     "documentArrayItem": {
-      "$comment": "The element schema of a typed array: one scalar, an integer, a number, a string, a boolean, or a byte array (byteArray: true), which with the identifier contentMediaType is an identifier. Objects and arrays of arrays are refused. An element carries no position, requiredSince, refersTo, uniqueItems, const or examples of its own (a one-value enum does what const would, and an update can widen it; examples annotate nothing an element needs). An enum's members are values of the element type (a byte array or identifier element takes none), and an integer element's minimum and maximum are integers; the parser reads them so random documents stay inside them",
+      "$comment": "The element schema of a typed array: one scalar, an integer, a number, a string, a boolean, or a byte array (byteArray: true), which with the identifier contentMediaType is an identifier. Objects and arrays of arrays are refused. An element carries no position, requiredSince, uniqueItems, const or examples of its own (a one-value enum does what const would, and an update can widen it; examples annotate nothing an element needs). An identifier element may carry a refersTo, which every element then declares. An enum's members are values of the element type (a byte array or identifier element takes none), and an integer element's minimum and maximum are integers; the parser reads them so random documents stay inside them",
       "type": "object",
       "properties": {
         "$comment": {
@@ -760,13 +760,51 @@
           "type": "string",
           "minLength": 1,
           "maxLength": 256
+        },
+        "refersTo": {
+          "description": "Only on identifier elements: what every element refers to, the refersTo declaration of an identifier property with the same keys and the same checks, except that identityPublicKey is refused: its keyIdProperty names one sibling key id, which cannot pair with many elements. When a document is created or replaced each element is checked as a single reference is, and the first one that fails refuses the write, its error naming the element by its list path (reasons[2] for the third). A propertyAgreement's referring side is still a property of the referring document or its $ownerId, the same for every element, and its referenced side a property of that element's referenced document. The declaration belongs on the items, not on the array. Available from protocol version 14.",
+          "$ref": "#/$defs/documentSchema/properties/refersTo",
+          "properties": {
+            "type": {
+              "not": {
+                "const": "identityPublicKey"
+              }
+            }
+          }
         }
       },
       "required": [
         "type"
       ],
       "additionalProperties": false,
       "dependentSchemas": {
+        "refersTo": {
+          "description": "refersTo is only allowed on identifier elements",
+          "properties": {
+            "type": {
+              "const": "array"
+            },
+            "byteArray": {
+              "const": true
+            },
+            "contentMediaType": {
+              "const": "application/x.dash.dpp.identifier"
+            },
+            "minItems": {
+              "const": 32
+            },
+            "maxItems": {
+              "const": 32
+            }
+          },
+          "required": [
+            "type",
+            "byteArray",
+            "contentMediaType",
+            "minItems",
+            "maxItems"
+          ]
+        },
         "distinctFrom": {
           "description": "distinctFrom is only allowed on identifier elements",
           "properties": {

packages/rs-dpp/src/data_contract/document_type/class_methods/create_document_types_from_document_schemas/v1/mod.rs

@@ -90,7 +90,9 @@ impl DocumentType {
 
         for (name, document_type) in &contract_document_types {
             for (path, property) in document_type.as_ref().flattened_properties() {
-                // Both forms of the key reference carry the same requirements
+                // Both forms of the key reference carry the same requirements.
+                // Only a scalar property can hold either: the typed array
+                // parser refuses identityPublicKey on an element
                 let key_requirements = match &property.property_type {
                     DocumentPropertyType::IdentifierWithReference(
                         DocumentPropertyReferenceTarget::IdentityPublicKey {

packages/rs-dpp/src/data_contract/document_type/class_methods/parse_typed_array/mod.rs

@@ -16,7 +16,9 @@ mod v0;
 /// Returns `None` for every other property, a byte array included, which the
 /// caller leaves to `DocumentPropertyType::try_from_value_map`. The element
 /// schema is parsed by that same scalar parser with the property's `options`,
-/// so an element has the type a scalar property of its schema would have.
+/// so an element has the type a scalar property of its schema would have,
+/// and a `refersTo` on identifier elements is folded in by the same
+/// `apply_property_reference` a scalar identifier goes through.
 ///
 /// Versioned on `parse_typed_array` in the platform version's document type
 /// schema versions. `None` selects the behavior of the versions that predate
@@ -35,7 +37,7 @@ pub(crate) fn parse_typed_array(
         .parse_typed_array
     {
         None => Ok(None),
-        Some(0) => v0::parse_typed_array_v0(inner_properties, options),
+        Some(0) => v0::parse_typed_array_v0(inner_properties, options, platform_version),
         Some(version) => Err(DataContractError::Unsupported(format!(
             "parse_typed_array version {version} is not supported"
         ))),