fix(dpp)!: refuse lookup key sources inside a transient object (PV14)

book/src/data-model/documents.md

@@ -324,7 +324,7 @@ A `deletableDocument` reference takes no `lookup`. Once the document a key found
 What is checked when the contract enters the chain, on registration and on update:
 
 - `lookup` is only allowed on `permanentDocument` references (meta-schema v3 and the parser, `apply_property_reference` 0), on the property or on the `items` of a typed array.
-- Each property a key reads must exist on the referring type, be required (and so must every object around it), not be transient, and hold a single value, so a lookup never runs with a missing key part and a reader can assemble the same key from the stored document. A key that reads `"$ownerId"` needs a referring type whose documents can be neither transferred nor traded: the reference is judged when the document is written, and a transfer or purchase would move the writer part of its key without a write. These are properties of the referring type alone and are checked on every parse (generation 3).
+- Each property a key reads must exist on the referring type, be required (and so must every object around it), not be transient nor sit inside a transient object, and hold a single value, so a lookup never runs with a missing key part and a reader can assemble the same key from the stored document. A key that reads `"$ownerId"` needs a referring type whose documents can be neither transferred nor traded: the reference is judged when the document is written, and a transfer or purchase would move the writer part of its key without a write. These are properties of the referring type alone and are checked on every parse (generation 3).
 - The index must exist and be unique, so the key finds at most one document; it may not bucket a timestamp with `timeRange`, and the referenced type may not be `indexOnly`. `keys` must cover each property of the index exactly once and nothing else, and each source must hold the same kind of value as the index property it fills (the rule of `propertyAgreement`, `DocumentPropertyType::value_kind`).
 - The key must stay with the document it found, or the reference could dangle without the document being deleted: every schema property of the index must be fixed once written (the referenced type is immutable, or the property, or the top-level object holding it, is listed under `immutable`), `$ownerId` is only a key part on a type whose documents can be neither transferred nor traded, and the update and transfer times are refused where a replace, transfer or purchase moves them. `$id`, `$creatorId` and the creation times are always fixed.
 - A changed, added or removed `lookup` is an incompatible schema change on update, like the rest of a `refersTo`.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -289,7 +289,7 @@
               }
             },
             "lookup": {
-              "description": "permanentDocument references only (a key into a deletable type could find a new document once the one it found is deleted, where an id is produced at most once): the property's value is not the referenced document's id. The referenced document is the one the named unique index of the referenced document type finds for a key assembled from the referring document, and the reference holds if that document exists. index names an index of the referenced document type that is unique, carries no timeRange and does not belong to an indexOnly type; its key must stay with the document it found (every schema property of the index fixed by an immutable document type or the immutable list, $ownerId only on a type whose documents can be neither transferred nor traded), so the reference never dangles; keys maps every property of that index (by its name on the referenced side, system ones such as $ownerId included), in any order, to where its value comes from: a property path of the referring document type, \"$ownerId\" for the referring document's owner, or \".\" for the value of the property carrying the reference (on an array element, the element), which must appear exactly once. A property a key reads must be required (and so must every object around it), not transient, and hold the same kind of value as its index property, so a lookup never runs with a missing key part; all of this is validated at contract registration. When the referring document is created or replaced, consensus queries the index for the assembled key, billed as a document fetch, and refuses the write with ReferencedEntityNotFoundError (40120) if it finds no document; propertyAgreement pairs are checked against the document found. A replace re-validates the reference when a property a key reads changed, and on every replace when a key reads $ownerId",
+              "description": "permanentDocument references only (a key into a deletable type could find a new document once the one it found is deleted, where an id is produced at most once): the property's value is not the referenced document's id. The referenced document is the one the named unique index of the referenced document type finds for a key assembled from the referring document, and the reference holds if that document exists. index names an index of the referenced document type that is unique, carries no timeRange and does not belong to an indexOnly type; its key must stay with the document it found (every schema property of the index fixed by an immutable document type or the immutable list, $ownerId only on a type whose documents can be neither transferred nor traded), so the reference never dangles; keys maps every property of that index (by its name on the referenced side, system ones such as $ownerId included), in any order, to where its value comes from: a property path of the referring document type, \"$ownerId\" for the referring document's owner, or \".\" for the value of the property carrying the reference (on an array element, the element), which must appear exactly once. A property a key reads must be required (and so must every object around it), not transient nor inside a transient object, and hold the same kind of value as its index property, so a lookup never runs with a missing key part; all of this is validated at contract registration. When the referring document is created or replaced, consensus queries the index for the assembled key, billed as a document fetch, and refuses the write with ReferencedEntityNotFoundError (40120) if it finds no document; propertyAgreement pairs are checked against the document found. A replace re-validates the reference when a property a key reads changed, and on every replace when a key reads $ownerId",
               "type": "object",
               "properties": {
                 "index": {

packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/reference_lookup_tests.rs

@@ -368,6 +368,37 @@ fn should_refuse_an_optional_transient_or_missing_lookup_source_property() {
     contract(nested).expect("a required leaf of a required object is a valid source");
 }
 
+/// `transient` names the object, not the leaf the key reads, and the whole
+/// object is stripped before storage, so a reader could never reassemble the
+/// key from the stored document.
+#[test]
+fn should_refuse_a_lookup_source_inside_a_transient_required_object() {
+    let mut schema = charter_contract(permanent_join_request(json!({
+        "index": "bySubmittedCharter",
+        "keys": { "submittedCharterId": "meta.charterId", "$ownerId": "." }
+    })));
+    let elected_charter = &mut schema["documentSchemas"]["electedCharter"];
+    elected_charter["properties"]["meta"] = json!({
+        "type": "object",
+        "position": 4,
+        "properties": { "charterId": identifier(0) },
+        "required": ["charterId"],
+        "additionalProperties": false
+    });
+    elected_charter["required"] = json!(["submittedCharterId", "title", "meta"]);
+    contract(schema.clone()).expect("a stored, required leaf of a required object is a source");
+
+    schema["documentSchemas"]["electedCharter"]["transient"] = json!(["meta"]);
+    for full_validation in [true, false] {
+        assert_refused(
+            contract_on(schema.clone(), full_validation, PlatformVersion::latest()),
+            "document type \"electedCharter\" property \"memberId\" refersTo lookup: key \
+             \"submittedCharterId\" reads \"meta.charterId\", which is transient or inside a \
+             transient object",
+        );
+    }
+}
+
 #[test]
 fn should_refuse_a_lookup_into_a_document_type_that_can_move_the_key() {
     let mutable = |schema: &mut serde_json::Value| {

packages/rs-dpp/src/data_contract/document_type/property/reference_lookup.rs

@@ -192,10 +192,10 @@ impl DocumentReferenceLookup {
                     "key \"{index_property}\" reads \"{path}\", which is not a single value"
                 ));
             }
-            if declaring.transient_fields().contains(path) {
+            if is_transient(declaring, path) {
                 return Some(format!(
-                    "key \"{index_property}\" reads \"{path}\", which is transient: the key must \
-                     be readable from the stored document"
+                    "key \"{index_property}\" reads \"{path}\", which is transient or inside a \
+                     transient object: the key must be readable from the stored document"
                 ));
             }
             // A required leaf inside an optional object is only present when the
@@ -400,9 +400,6 @@ impl DocumentReferenceLookup {
     }
 }
 
-/// The kind of value an index property of `document_type` holds: a system
-/// property's fixed type, or the schema property's. `None` when the name is
-/// neither.
 /// Whether a document of `document_type` can change owner after it was
 /// written, by a transfer or a purchase. Both flags are immutable on contract
 /// update, so the answer holds for good.
@@ -411,6 +408,21 @@ fn owner_can_change(document_type: DocumentTypeRef) -> bool {
         || document_type.trade_mode() != TradeMode::None
 }
 
+/// Whether the property at `path` of `document_type`, or an object around it,
+/// is transient: either way its value is never stored. `transient_fields()`
+/// holds the paths as declared, so a leaf of a transient object is found only
+/// through the object's path, a prefix of its own.
+fn is_transient(document_type: DocumentTypeRef, path: &str) -> bool {
+    let transient_fields = document_type.transient_fields();
+    path.match_indices('.')
+        .map(|(end, _)| &path[..end])
+        .chain(std::iter::once(path))
+        .any(|prefix| transient_fields.contains(prefix))
+}
+
+/// The kind of value an index property of `document_type` holds: a system
+/// property's fixed type, or the schema property's. `None` when the name is
+/// neither.
 fn index_property_value_kind(
     document_type: DocumentTypeRef,
     name: &str,