diff --git a/book/src/drive/index-only-document-types.md b/book/src/drive/index-only-document-types.md index 98903aabe97..34b72a4454a 100644 --- a/book/src/drive/index-only-document-types.md +++ b/book/src/drive/index-only-document-types.md @@ -28,11 +28,14 @@ stores nothing in primary storage. The index entries ARE the rows: → Item(, flags) ``` -The **terminal** — a per-index keyword defaulting to `$ownerId`, or any -refersTo-typed identifier property (identity, contract, token, permanent -or deletable document) — is the member key, sitting exactly where a normal non-unique -index keys by document id; the element is an `Item` instead of a -`Reference` because there is nothing to point at. The `0` storage marker, +The **terminal** is the member key, sitting exactly where a normal +non-unique index keys by document id; the element is an `Item` instead of +a `Reference` because there is nothing to point at. It is a per-index +keyword defaulting to `$ownerId`. It may name any schema property a prefix +position could carry (an identifier with or without a `refersTo`, a +bounded byte array or string, an integer, a boolean, a date), or an +ordered **list** of such properties (a *composite* terminal, whose member +key is their encoded values concatenated). The `0` storage marker, value-tree types, and the count/sum/ranked tree derivation are byte-identical to the ordinary non-unique layout, which is what lets the protocol v14 ranked machinery (see @@ -41,6 +44,87 @@ types unchanged: "the five most-liked posts in `#dash`" is an O(log n + k) read with an O(log n + k) proof, and Items count in count/ranked trees exactly as References do. +The member key is the terminal value's **tree-key encoding**, produced by +the same functions the prefix levels use (the walkers and probes through +`get_raw_for_document_type`, queries and executed proofs through +`serialize_value_for_key`, synthesis through `decode_value_for_tree_keys`), +so nothing about it is specific to a 32-byte identifier: a 33-byte public +key, a short string or an integer keys the `0` bucket exactly as it would +key a prefix level, and fee estimation sizes the member key by the +terminal property's declared bound (`index_only_terminal_max_key_size`) +rather than by a fixed 32. Structural uniqueness spans the terminal value: +one entry per (prefix values, terminal value), so two documents by one +owner that differ only in a scalar terminal are two entries under the +same prefix. + +**Composite terminals.** `"terminal": ["kind", "$ownerId"]` keys the +member by `encode(kind) ‖ owner`. Every component but the last must be +fixed width (a byte array with `minItems == maxItems`, an identifier, an +integer, a boolean, a date), so equality on the leading components is a +clean key range and synthesis can split the key back; a string can only +be the last component; the whole key is capped at 255 bytes. Uniqueness +spans the whole key. Queries bind the components in order: equality +clauses on the leading ones, then at most one range or `in` clause on the +next (ordered by it), nothing on the rest. The lowering pads the bound +prefix with `0xFF` to the key cap for the upper bound of "every key under +this prefix", and addresses the key itself when the bound component is +the last one. After equality-bound components are ignored, `orderBy` must +start at the first remaining component and follow component order without +gaps, with the same direction for every listed component. A single member-key +walk cannot sort by a later component alone or mix ascending and descending +components. + +**Flat indexes.** An index with no `properties` at all is *flat*: its +entries live directly under a level of their own, keyed by a zero byte +followed by each terminal component name preceded by a zero byte +(`"\0appEphemeralPubKeyHash\0$ownerId"`), which no property-name tree can +collide with since property names never contain a zero byte. This level key, +including its separators, must also fit within 255 bytes: + +```text +[DataContractDocuments, contract_id, 1, , "\0\0…", 0, ] + → Item( [‖ ], flags) +``` + +The flat level is registration-time structure, created with the +property-name trees and kept when the last entry goes (the prune stops at +its `0` bucket, as on a preallocated index), so every entry costs the same. +There is no prefix level for an aggregate, a ranking, a time grid, a skip +trigger or a preallocation to apply to, so a flat index admits none of +those keywords. A clause-free query on a type with a flat index scans the +flat level (every other indexOnly type refuses the by-id shape). Non-proof +responses require this index to cover every property, including optional +ones, just as filtered queries do; otherwise use a proved projection. + +**The entry payload.** `entryPayload: ["walletEphemeralPubKey", +"encryptedPayload"]` on the document type names top-level properties that +live in no index: every entry's item carries them after the 32-byte row +commitment, each length-framed (`u16` big-endian), in property-name order: +the type's value slot. Byte arrays store their raw bytes and strings store +UTF-8; other scalars use their tree-key encoding. The length frame preserves +empty byte arrays and strings without null sentinels, and distinguishes an +empty string from a NUL string. A payload +property must be required, scalar and bounded (the sum of the bounds is +capped by the field value limit), and appears in no index as a property +or a terminal component. It is still committed (the commitment hashes +every present property, a payload value through the uncapped payload +encoding), so the delete probes and the executed-transition verifier keep +comparing the item's first 32 bytes only, and synthesis decodes the rest +of the proved element. With more than one index the payload rides in +every entry; fee estimation sizes the item by the commitment plus the +payload bound. Together, a flat composite terminal and an entry payload +make a key-value table: + +```json +"indices": [{ "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] }], +"entryPayload": ["walletEphemeralPubKey", "encryptedPayload"] +``` + +lands at `[…, "\0appEphemeralPubKeyHash\0$ownerId", 0, hash ‖ owner] → +Item(commitment ‖ len ‖ ciphertext ‖ len ‖ wallet key)`, and a query on +the hash returns every responder's owner id with the payload decoded off +the item, as one proof. + **`timeRange` buckets** compose too: a bucketed indexOnly index writes one commitment entry per containing bucket under the grid-qualified level, exactly as stored types do — the walkers' bucket fan-out, the @@ -103,7 +187,9 @@ aggregate keywords follow: | every non-trigger property appears in ≥ 1 **non-skip** index (prefix or terminal) | only indexed values exist, and a skip index carries no value for trigger-absent documents — covered only there, a property would be validated and committed yet written nowhere | | **every index embeds `$ownerId`** (prefix or terminal) | entries are self-authorizing: a delete computed with owner = signer can only ever address the signer's own entries | | ≥ 1 index is `$createdAt`-free AND non-`skipIfAbsent` — the **proof index** | executed-transition proofs locate entries from the transition's values alone: they can neither reproduce a block timestamp nor anchor on an entry that may not exist | -| terminal is `$ownerId` or a single-id refersTo property | the member key must alone be a referable entity id (`identityPublicKey` is compound and rejected) | +| every terminal component is `$ownerId` or a schema property passing the indexed-shape limits (no arrays or objects; byte arrays ≤ 255 bytes, strings ≤ 63 characters); every component but the last is fixed width; the whole key ≤ 255 bytes | the member key is the components' tree-key encodings concatenated, derived by the same functions the prefix levels use; a leading component must be splittable back and rangeable; grovedb caps keys at 255 bytes; other system properties are refused because the `$createdAt` rules walk the prefix properties | +| a flat index (no `properties`) admits no countable / summable / ranked / `timeRange` / `skipIfAbsent` / `preallocated` keyword | there is no prefix level for them to apply to | +| every `entryPayload` property is a required, bounded, top-level scalar in no index | the entry value has no representation for an absent property, estimation sizes the item by the bounds, and a property is either a key or a value | | indexed `$createdAt` requires `$createdAt` in `required` | creation only assigns timestamps for required system times | | `documentsMutable: false`, no transfers/trading/history/transient | no stored row, no revision | | non-unique, non-contested, `nullSearchable` default | v1 scope | diff --git a/packages/kotlin-sdk/KotlinExampleApp/app/src/main/java/org/dashfoundation/example/ui/contracts/ContractJson.kt b/packages/kotlin-sdk/KotlinExampleApp/app/src/main/java/org/dashfoundation/example/ui/contracts/ContractJson.kt index 5d28ec01338..5ea90087fc2 100644 --- a/packages/kotlin-sdk/KotlinExampleApp/app/src/main/java/org/dashfoundation/example/ui/contracts/ContractJson.kt +++ b/packages/kotlin-sdk/KotlinExampleApp/app/src/main/java/org/dashfoundation/example/ui/contracts/ContractJson.kt @@ -195,10 +195,17 @@ internal fun indexAxisDescriptors(index: JsonObject): List { } /** - * The index's member-key property on an indexOnly document type: the - * declared `terminal`, defaulting to `$ownerId` exactly as DPP - * normalizes an omitted terminal. `null` on stored (non-indexOnly) - * document types, where entries are keyed by document id. + * The index's member key on an indexOnly document type: the declared + * `terminal` (a property name, or the ordered component names of a + * composite terminal joined with ` ‖ ` for display), defaulting to + * `$ownerId` exactly as DPP normalizes an omitted terminal. `null` on + * stored (non-indexOnly) document types, where entries are keyed by + * document id. */ internal fun indexTerminal(index: JsonObject, indexOnly: Boolean): String? = - index.stringField("terminal") ?: if (indexOnly) "\$ownerId" else null + index.stringField("terminal") + ?: (index["terminal"] as? JsonArray) + ?.mapNotNull { (it as? JsonPrimitive)?.content } + ?.takeIf { it.isNotEmpty() } + ?.joinToString(" ‖ ") + ?: if (indexOnly) "\$ownerId" else null diff --git a/packages/kotlin-sdk/KotlinExampleApp/app/src/test/java/org/dashfoundation/example/ui/contracts/IndexKeywordDescriptorsTest.kt b/packages/kotlin-sdk/KotlinExampleApp/app/src/test/java/org/dashfoundation/example/ui/contracts/IndexKeywordDescriptorsTest.kt index 75122722c35..f3d0f5687a0 100644 --- a/packages/kotlin-sdk/KotlinExampleApp/app/src/test/java/org/dashfoundation/example/ui/contracts/IndexKeywordDescriptorsTest.kt +++ b/packages/kotlin-sdk/KotlinExampleApp/app/src/test/java/org/dashfoundation/example/ui/contracts/IndexKeywordDescriptorsTest.kt @@ -1,6 +1,8 @@ package org.dashfoundation.example.ui.contracts import kotlinx.serialization.json.JsonObject +import kotlinx.serialization.json.add +import kotlinx.serialization.json.buildJsonArray import kotlinx.serialization.json.buildJsonObject import kotlinx.serialization.json.put import org.junit.Assert.assertEquals @@ -92,5 +94,10 @@ class IndexKeywordDescriptorsTest { val declared = buildJsonObject { put("terminal", "postId") } assertEquals("postId", indexTerminal(declared, indexOnly = true)) assertEquals("postId", indexTerminal(declared, indexOnly = false)) + + val composite = buildJsonObject { + put("terminal", buildJsonArray { add("postId"); add("\$ownerId") }) + } + assertEquals("postId ‖ \$ownerId", indexTerminal(composite, indexOnly = true)) } } diff --git a/packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json b/packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json index b20906d47f3..93621398d5b 100644 --- a/packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json +++ b/packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json @@ -721,10 +721,25 @@ "description": "Buckets the first index property's timestamp into fixed-length, regularly-spaced (possibly overlapping) time ranges. The window parameters (`range`, `step`, `phase`) are declared in seconds, since a bucket is selected from block time and the target block interval is five seconds; the stored key is the range start as a u64 millisecond timestamp, so it stays directly comparable to the source timestamp it buckets. Enables trending/leaderboard queries within the newest/oldest active range. A system-timestamp source must be listed in the document type's required fields. Several indexes may bucket the same timestamp with different grids — each grid gets its own index subtree, keyed by the property name qualified with the grid parameters. Available from protocol version 14." }, "terminal": { - "type": "string", - "minLength": 1, - "maxLength": 256, - "description": "Only on indexOnly document types: names the property whose value is this index entry's member key — the docId-analog terminal key under the index's storage marker, stored as an Item instead of a Reference because there is no primary-storage row. Either \"$ownerId\" (the default when omitted) or an identifier property carrying a refersTo declaration (identity, contract, token, permanentDocument, or deletableDocument). Must not repeat one of the index's listed properties. Available from protocol version 14." + "oneOf": [ + { + "type": "string", + "minLength": 1, + "maxLength": 256 + }, + { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 256 + }, + "minItems": 1, + "maxItems": 10, + "uniqueItems": true + } + ], + "description": "Only on indexOnly document types: the property, or the ordered list of properties (a composite terminal), whose encoded values, concatenated, are this index entry's member key — the docId-analog terminal key under the index's storage marker, stored as an Item instead of a Reference because there is no primary-storage row. Each component is \"$ownerId\" (the default when the keyword is omitted) or any schema property a prefix position could carry (no arrays or objects; byte arrays of at most 255 bytes, strings of at most 63 characters); every component but the last must be fixed width (a byte array with minItems equal to maxItems, an identifier, an integer, a boolean or a date), and the whole key at most 255 bytes. One entry exists per (prefix values, terminal values). No other system property may be a component, and none may repeat one of the index's listed properties. An index with no properties is a flat index keyed by its terminal alone, admitting no aggregate, ranking, timeRange, skipIfAbsent or preallocated keyword. Available from protocol version 14." }, "preallocated": { "type": "boolean", @@ -736,7 +751,6 @@ } }, "required": [ - "properties", "name" ], "dependentRequired": { @@ -903,6 +917,18 @@ "type": "boolean", "description": "When true, documents of this type are never written to primary storage: the index entries are the rows, each terminating in an Item keyed by the index's `terminal` property instead of a Reference keyed by the document id. Only what is in the indexes exists and is recoverable. Requires: every property required and appearing in at least one index (except a `skipIfAbsent` index's optional first property), $ownerId in at least one index (as a property or terminal), documentsMutable: false, no transfers/trading/history/transient properties, and no doctype-level aggregate keywords (use the index-level count flags). Available from protocol version 14." }, + "entryPayload": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 64 + }, + "minItems": 1, + "maxItems": 16, + "uniqueItems": true, + "description": "Only on indexOnly document types: the top-level properties stored in every entry's value, after the 32-byte row commitment, instead of in a key — the type's value slot. Each is length-framed in property-name order and recovered by decoding the proved element: raw bytes for byte arrays, UTF-8 for strings, and tree-key encoding for other scalars. Empty byte arrays and strings remain distinct from null, and an empty string remains distinct from a NUL string. Listed properties must be required, bounded (maxItems on byte arrays, maxLength on strings), scalar, and must not appear in any index as a property or a terminal component; their summed bounds are capped by the field value limit. With more than one index the payload rides in every entry. Fixed when the document type is created. Available from protocol version 14." + }, "actionFees": { "type": "object", "description": "A fixed fee in credits charged, on top of the gas, for actions on documents of this type, split between the contract's owner pot and its moderators pot (each paid out by a ContractFeeClaim state transition). Whoever pays the gas of the action pays its fee. A transition on a priced action must name the declared owner and moderators amounts, and for feeMultiplier pricing the fee multiplier its signer knew with the increase in percent they accept, in $actionFeeAgreement; it is refused without one (DocumentActionFeeAgreementNotSetError, code 40132), with other amounts or another pricing (code 40133), or when the epoch's multiplier rose beyond the accepted increase (code 40134). At least one action must be priced and a priced action must charge something. Fixed when the document type is published: a contract update cannot add, change or remove the fees of an existing document type. Available from protocol version 14.", diff --git a/packages/rs-dpp/src/data_contract/document_type/accessors/mod.rs b/packages/rs-dpp/src/data_contract/document_type/accessors/mod.rs index 0e34a7b57d2..589b295f98c 100644 --- a/packages/rs-dpp/src/data_contract/document_type/accessors/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/accessors/mod.rs @@ -945,6 +945,7 @@ impl DocumentTypeV1Getters for DocumentTypeMutRef<'_> { /// predate the keyword: V0 and V1 have no field to borrow from, and the /// getter hands out a reference. static NO_IMMUTABLE_FIELDS: BTreeSet = BTreeSet::new(); +static NO_ENTRY_PAYLOAD: BTreeSet = BTreeSet::new(); impl DocumentTypeV2Getters for DocumentType { fn documents_countable(&self) -> bool { @@ -987,6 +988,15 @@ impl DocumentTypeV2Getters for DocumentType { } } + /// The entry-payload property names of an indexOnly type (empty before V2). + fn entry_payload(&self) -> &BTreeSet { + match self { + DocumentType::V0(_) => &NO_ENTRY_PAYLOAD, + DocumentType::V1(_) => &NO_ENTRY_PAYLOAD, + DocumentType::V2(v2) => v2.entry_payload(), + } + } + fn documents_can_be_deleted_by_moderators(&self) -> bool { match self { DocumentType::V0(_) => false, @@ -1103,6 +1113,15 @@ impl DocumentTypeV2Getters for DocumentTypeRef<'_> { } } + /// The entry-payload property names of an indexOnly type (empty before V2). + fn entry_payload(&self) -> &BTreeSet { + match self { + DocumentTypeRef::V0(_) => &NO_ENTRY_PAYLOAD, + DocumentTypeRef::V1(_) => &NO_ENTRY_PAYLOAD, + DocumentTypeRef::V2(v2) => v2.entry_payload(), + } + } + fn documents_can_be_deleted_by_moderators(&self) -> bool { match self { DocumentTypeRef::V0(_) => false, @@ -1185,6 +1204,15 @@ impl DocumentTypeV2Getters for DocumentTypeMutRef<'_> { } } + /// The entry-payload property names of an indexOnly type (empty before V2). + fn entry_payload(&self) -> &BTreeSet { + match self { + DocumentTypeMutRef::V0(_) => &NO_ENTRY_PAYLOAD, + DocumentTypeMutRef::V1(_) => &NO_ENTRY_PAYLOAD, + DocumentTypeMutRef::V2(v2) => v2.entry_payload(), + } + } + fn documents_can_be_deleted_by_moderators(&self) -> bool { match self { DocumentTypeMutRef::V0(_) => false, diff --git a/packages/rs-dpp/src/data_contract/document_type/accessors/v2/mod.rs b/packages/rs-dpp/src/data_contract/document_type/accessors/v2/mod.rs index f492facd064..063ffe3fd54 100644 --- a/packages/rs-dpp/src/data_contract/document_type/accessors/v2/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/accessors/v2/mod.rs @@ -33,6 +33,10 @@ pub trait DocumentTypeV2Getters { /// each terminating in an `Item` keyed by the index's `terminal` /// property. Only what is in the indexes exists and is recoverable. fn index_only(&self) -> bool; + /// On an indexOnly type, the top-level properties stored in every entry's + /// value after the row commitment (`entryPayload`), in name order; empty + /// elsewhere. + fn entry_payload(&self) -> &BTreeSet; /// Returns whether the contract's moderators may delete documents of this /// type (the `canBeDeletedByModerators` keyword, protocol version 14). diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs index 57de57dbcd1..cc6a8039657 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs @@ -71,8 +71,7 @@ use crate::consensus::basic::data_contract::{ }; #[cfg(feature = "validation")] use crate::consensus::basic::data_contract::{ - DuplicateIndexNameError, InvalidIndexPropertyTypeError, InvalidIndexedPropertyConstraintError, - SystemPropertyIndexAlreadyPresentError, UndefinedIndexPropertyError, + DuplicateIndexNameError, SystemPropertyIndexAlreadyPresentError, UndefinedIndexPropertyError, UniqueIndicesLimitReachedError, }; #[cfg(feature = "validation")] @@ -100,9 +99,10 @@ use jsonschema::JSONSchema; use std::collections::HashSet; #[cfg(feature = "validation")] -use super::{ - MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH, MAX_INDEXED_STRING_PROPERTY_LENGTH, - NOT_ALLOWED_SYSTEM_PROPERTIES, +use super::NOT_ALLOWED_SYSTEM_PROPERTIES; +use super::{MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH, MAX_INDEXED_STRING_PROPERTY_LENGTH}; +use crate::consensus::basic::data_contract::{ + InvalidIndexPropertyTypeError, InvalidIndexedPropertyConstraintError, }; /// RANKED: the extra index-property check a generation runs before the generic @@ -1161,7 +1161,7 @@ fn parse_indices( use crate::document::property_names::OWNER_ID; for index in indices.values_mut() { if index.terminal.is_none() { - index.terminal = Some(OWNER_ID.to_string()); + index.terminal = Some(vec![OWNER_ID.to_string()]); } } } @@ -1288,68 +1288,90 @@ fn validate_index_properties( ctx.platform_version, )?; - // Validate indexed property type - match &property_definition.property_type { - // Array and objects aren't supported for indexing yet - DocumentPropertyType::Array(_) - | DocumentPropertyType::Object(_) - | DocumentPropertyType::VariableTypeArray(_) => { - Err(ProtocolError::ConsensusError(Box::new( - InvalidIndexPropertyTypeError::new( - ctx.name.to_owned(), - index.name.to_owned(), - index_property.name.to_owned(), - property_definition.property_type.name(), - ) - .into(), - ))) - } - // Indexed byte array size must be limited - DocumentPropertyType::ByteArray(sizes) - if sizes.max_size.is_none() - || sizes.max_size.unwrap() > MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH => - { - Err(ProtocolError::ConsensusError(Box::new( - InvalidIndexedPropertyConstraintError::new( - ctx.name.to_owned(), - index.name.to_owned(), - index_property.name.to_owned(), - "maxItems".to_string(), - format!( - "should be less or equal {}", - MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH - ), - ) - .into(), - ))) - } - // Indexed string length must be limited - DocumentPropertyType::String(sizes) - if sizes.max_length.is_none() - || sizes.max_length.unwrap() > MAX_INDEXED_STRING_PROPERTY_LENGTH => - { - Err(ProtocolError::ConsensusError(Box::new( - InvalidIndexedPropertyConstraintError::new( - ctx.name.to_owned(), - index.name.to_owned(), - index_property.name.to_owned(), - "maxLength".to_string(), - format!( - "should be less or equal {}", - MAX_INDEXED_STRING_PROPERTY_LENGTH - ), - ) - .into(), - ))) - } - _ => Ok(()), - } + // The shape limits every indexed value carries as a grovedb key, + // shared with an indexOnly index's terminal. + check_indexable_property_shape( + ctx.name, + &index.name, + &index_property.name, + &property_definition.property_type, + ) } else { Ok(()) } }) } +/// The shape checks a property must pass to be indexed, shared by the +/// prefix positions of an index and an indexOnly index's terminal: the +/// encoded value becomes a grovedb key, so arrays and objects are refused +/// and byte arrays and strings must be bounded (grovedb caps keys at 255 +/// bytes; the string bound is in characters, each at most four bytes). +fn check_indexable_property_shape( + document_type_name: &str, + index_name: &str, + property_name: &str, + property_type: &DocumentPropertyType, +) -> Result<(), ProtocolError> { + match property_type { + // Array and objects aren't supported for indexing yet + DocumentPropertyType::Array(_) + | DocumentPropertyType::Object(_) + | DocumentPropertyType::VariableTypeArray(_) => { + Err(ProtocolError::ConsensusError(Box::new( + InvalidIndexPropertyTypeError::new( + document_type_name.to_owned(), + index_name.to_owned(), + property_name.to_owned(), + property_type.name(), + ) + .into(), + ))) + } + // Indexed byte array size must be limited + DocumentPropertyType::ByteArray(sizes) + if sizes + .max_size + .is_none_or(|max_size| max_size > MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH) => + { + Err(ProtocolError::ConsensusError(Box::new( + InvalidIndexedPropertyConstraintError::new( + document_type_name.to_owned(), + index_name.to_owned(), + property_name.to_owned(), + "maxItems".to_string(), + format!( + "should be less or equal {}", + MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH + ), + ) + .into(), + ))) + } + // Indexed string length must be limited + DocumentPropertyType::String(sizes) + if sizes + .max_length + .is_none_or(|max_length| max_length > MAX_INDEXED_STRING_PROPERTY_LENGTH) => + { + Err(ProtocolError::ConsensusError(Box::new( + InvalidIndexedPropertyConstraintError::new( + document_type_name.to_owned(), + index_name.to_owned(), + property_name.to_owned(), + "maxLength".to_string(), + format!( + "should be less or equal {}", + MAX_INDEXED_STRING_PROPERTY_LENGTH + ), + ) + .into(), + ))) + } + _ => Ok(()), + } +} + /// The identifier and binary paths implied by the parsed properties, plus /// the security level and the encryption/decryption key requirements the /// schema asks for. @@ -2318,6 +2340,7 @@ pub(super) fn apply_index_only( document_type: &mut DocumentTypeV2, index_only: bool, name: &str, + platform_version: &PlatformVersion, ) -> Result<(), ProtocolError> { use crate::document::property_names::{CREATED_AT, OWNER_ID}; @@ -2341,6 +2364,29 @@ pub(super) fn apply_index_only( index_name, name, ))); } + // An index without properties is only meaningful as a flat indexOnly + // index (keyed by its terminal alone); on a stored type it would + // reach no level at all and index nothing. + if let Some((index_name, _)) = document_type + .indices + .iter() + .find(|(_, index)| index.properties.is_empty()) + { + return Err(structure_error(format!( + "index \"{}\" on document type \"{}\" has no properties: an index keyed by \ + a terminal alone (a flat index) is only allowed on an indexOnly document \ + type", + index_name, name, + ))); + } + if !document_type.entry_payload.is_empty() { + return Err(structure_error(format!( + "document type \"{}\" declares `entryPayload`, which is only allowed on \ + indexOnly document types (a stored document type keeps every property in \ + its primary row)", + name, + ))); + } // Same for `preallocated`: only an indexOnly index's trees are cheap // permanent structure whose member entries carry the data — on a // normal document type the trees hold references to stored rows and @@ -2452,15 +2498,109 @@ pub(super) fn apply_index_only( // structure's level info and the `Index` values below agree, and every // check here reads `Some`. + // ---- entry payload -------------------------------------------------- + // `entryPayload` names the type's value slot: top-level scalar + // properties stored in every entry's value, after the row commitment, + // instead of in a key. They are still committed (the commitment hashes + // every present property) and still required, but they sit in no + // index, so the every-property-indexed rule below exempts them. Each + // must be bounded, since fee estimation sizes the entry value by the + // sum of their bounds, and the sum is capped by the field value limit. + let mut payload_max_total: u32 = 0; + for payload_property in document_type.entry_payload.iter() { + let Some(property) = document_type.properties.get(payload_property) else { + return Err(structure_error(format!( + "entryPayload of indexOnly document type \"{}\" names \"{}\", which is not \ + a top-level property of the document type", + name, payload_property, + ))); + }; + if matches!( + property.property_type, + DocumentPropertyType::Object(_) + | DocumentPropertyType::Array(_) + | DocumentPropertyType::VariableTypeArray(_) + ) { + return Err(structure_error(format!( + "entryPayload property \"{}\" of indexOnly document type \"{}\" must be a \ + scalar (a byte array, string, integer, boolean, date or identifier): the \ + entry value is a flat concatenation of length-framed scalars", + payload_property, name, + ))); + } + let max_width = property + .property_type + .max_byte_size(platform_version)? + .unwrap_or(u16::MAX); + if max_width == u16::MAX { + return Err(structure_error(format!( + "entryPayload property \"{}\" of indexOnly document type \"{}\" must be \ + bounded (declare maxItems on a byte array or maxLength on a string): fee \ + estimation sizes every entry's value by the payload bounds", + payload_property, name, + ))); + } + // Two bytes of length frame per property. + payload_max_total += u32::from(max_width) + 2; + if !document_type.required_fields.contains(payload_property) { + return Err(structure_error(format!( + "entryPayload property \"{}\" of indexOnly document type \"{}\" must be \ + listed in `required`: the entry value has no representation for an absent \ + property", + payload_property, name, + ))); + } + if let Some((index_name, _)) = document_type.indices.iter().find(|(_, index)| { + index.terminal_contains(payload_property) + || index + .properties + .iter() + .any(|index_property| index_property.name == *payload_property) + }) { + return Err(structure_error(format!( + "entryPayload property \"{}\" of indexOnly document type \"{}\" also \ + appears in index \"{}\": a property is either a key (a prefix property or \ + a terminal component) or entry payload, never both", + payload_property, name, index_name, + ))); + } + } + if payload_max_total > platform_version.system_limits.max_field_value_size { + return Err(structure_error(format!( + "entryPayload of indexOnly document type \"{}\" may encode to {} bytes, over \ + the {}-byte cap on an entry's value", + name, payload_max_total, platform_version.system_limits.max_field_value_size, + ))); + } + // ---- per-index rules ------------------------------------------------ for (index_name, index) in document_type.indices.iter() { if index.properties.is_empty() { - return Err(structure_error(format!( - "index \"{}\" on indexOnly document type \"{}\" has no properties: an \ - indexOnly entry is `[…property values, 0, terminal value]`, so at least \ - one prefix property is required above the terminal", - index_name, name, - ))); + // FLAT index: no prefix levels, the entries live directly under + // a level keyed by the terminal's component names. There is no + // prefix level for an aggregate, a ranking, a time grid, a skip + // trigger or a preallocation to apply to, so none of those + // keywords is admitted on it. + if index.countable.is_countable() + || index.range_countable + || index.summable.is_some() + || index.range_summable + || index.ranked_countable + || !index.ranked_countable_at.is_empty() + || index.ranked_summable + || index.ranked_averageable + || index.time_range.is_some() + || index.skip_if_absent + || index.preallocated + { + return Err(structure_error(format!( + "index \"{}\" on indexOnly document type \"{}\" has no properties (a \ + flat index keyed by its terminal alone), so it admits no countable, \ + summable, ranked, timeRange, skipIfAbsent or preallocated keyword: \ + there is no prefix level for them to apply to", + index_name, name, + ))); + } } if index.unique { return Err(structure_error(format!( @@ -2558,67 +2698,143 @@ pub(super) fn apply_index_only( // (canonical property, i64-safe integer type, `required` // membership) run for every doctype, indexOnly included. - let terminal = index.terminal.as_deref().expect("normalized to Some above"); - - if index - .properties - .iter() - .any(|property| property.name == terminal) - { + let components = index.terminal_components(); + if components.is_empty() { return Err(structure_error(format!( - "index \"{}\" on indexOnly document type \"{}\" repeats its terminal \ - (\"{}\") in its properties: the terminal is the member key below the \ - listed properties, so listing it again would index the same dimension \ - twice", - index_name, name, terminal, + "index \"{}\" on indexOnly document type \"{}\" has no terminal after \ + normalization: internal parser error", + index_name, name, ))); } - // The terminal is the member key — it must be a referable entity id: - // the owner identity, or a property carrying a refersTo declaration - // whose value alone IS the referenced entity's id (identity, - // contract, token, or a document of either reference kind; a - // deletable document's entries simply outlive it, as the member key - // is an Item, not a Reference). `identityPublicKey` is deliberately - // NOT admitted: it is a - // compound reference — this property carries the identity id while a - // separate `keyIdProperty` carries the key id — so a terminal keyed - // by it would conflate references to different keys of the same - // identity. - if terminal != OWNER_ID { - use crate::data_contract::document_type::property::DocumentPropertyReferenceTarget; - match document_type.flattened_properties.get(terminal) { - Some(property) - if matches!( - property.property_type, - DocumentPropertyType::IdentifierWithReference( - DocumentPropertyReferenceTarget::Identity - | DocumentPropertyReferenceTarget::Contract - | DocumentPropertyReferenceTarget::Token - | DocumentPropertyReferenceTarget::PermanentDocument { .. } - | DocumentPropertyReferenceTarget::DeletableDocument { .. } - ) - ) => {} - Some(_) => { + for component in components { + if index + .properties + .iter() + .any(|property| property.name == *component) + { + return Err(structure_error(format!( + "index \"{}\" on indexOnly document type \"{}\" repeats its terminal \ + component (\"{}\") in its properties: the terminal is the member key \ + below the listed properties, so listing it again would index the same \ + dimension twice", + index_name, name, component, + ))); + } + } + + // The terminal is the member key: the encoded values of its + // components, concatenated in order. Any property a prefix position + // admits may serve as a component: every path derives the member + // key through the same tree-key encoding the prefix levels use (the + // walkers and probes via `get_raw_for_document_type`, queries and + // executed proofs via `serialize_value_for_key`, synthesis via + // `decode_value_for_tree_keys`), so a component needs no particular + // width or meaning — only the shape limits every indexed value + // carries. Structural uniqueness spans the whole key: one entry per + // (prefix values, terminal values). + // + // Every component but the last must be fixed width: a leading + // component is followed by more key bytes, and only a fixed-width + // encoding keeps equality on the leading components a clean key + // range (and lets synthesis split the key back). Strings are never + // fixed width (their bound counts characters, not bytes), so a + // string can only be the last component. + // + // System properties other than `$ownerId` are refused: `$createdAt` + // is the one other system value an indexOnly entry can carry, and + // the rules that reason about it (the proof-index selection, + // `required` membership, bucketing) all walk the prefix properties, + // so admitting it as a component would need each of them extended + // first. + let mut terminal_max_width: u32 = 0; + for (position, component) in components.iter().enumerate() { + let is_last = position + 1 == components.len(); + let max_width: u32 = if component == OWNER_ID { + 32 + } else { + if component.starts_with('$') { + return Err(structure_error(format!( + "terminal component \"{}\" of index \"{}\" on indexOnly document \ + type \"{}\" is a system property: only $ownerId may be a terminal \ + component (name a schema property, or list $createdAt among the \ + index's properties instead)", + component, index_name, name, + ))); + } + // A flat level is keyed by its component names, each behind + // a zero byte (`flat_level_key_for`); a name carrying one + // would alias another flat level or a property-name tree. + // The meta-schema's name pattern already excludes it for + // contracts entering the chain; this keeps the invariant + // explicit for every parse. + if component.contains('\0') { return Err(structure_error(format!( - "terminal \"{}\" of index \"{}\" on indexOnly document type \"{}\" \ - must be \"$ownerId\" or an identifier property with a refersTo \ - declaration targeting identity, contract, token, \ - permanentDocument, or deletableDocument: the terminal is the \ - entry's member key and must \ - alone be a referable entity id (an identityPublicKey reference is \ - compound — its key id lives in a separate property — and is not \ - admitted)", - terminal, index_name, name, + "terminal component \"{}\" of index \"{}\" on indexOnly document \ + type \"{}\" contains a zero byte, which the flat level key \ + reserves as its separator", + component.escape_default(), + index_name, + name, ))); } - None => { + let Some(property) = document_type.flattened_properties.get(component) else { return Err(structure_error(format!( - "terminal \"{}\" of index \"{}\" on indexOnly document type \"{}\" \ - does not name a property of the document type", - terminal, index_name, name, + "terminal component \"{}\" of index \"{}\" on indexOnly document \ + type \"{}\" does not name a property of the document type", + component, index_name, name, + ))); + }; + check_indexable_property_shape( + name, + index_name, + component, + &property.property_type, + )?; + let max_width = property + .property_type + .max_byte_size(platform_version)? + .unwrap_or(u16::MAX); + // Fixed width by the tree-key encoding itself: the same + // helper synthesis splits member keys with. + let fixed_width = property.property_type.fixed_tree_key_width().is_some(); + if !is_last && !fixed_width { + return Err(structure_error(format!( + "terminal component \"{}\" of index \"{}\" on indexOnly document \ + type \"{}\" is followed by another component but is not fixed \ + width: every component but the last must encode to a fixed number \ + of bytes (a byte array with minItems equal to maxItems, an \ + identifier, an integer, a boolean or a date); a string or a \ + variable-size byte array can only be the last component", + component, index_name, name, ))); } + u32::from(max_width) + }; + terminal_max_width += max_width; + } + if terminal_max_width > u32::from(MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH) { + return Err(structure_error(format!( + "the terminal of index \"{}\" on indexOnly document type \"{}\" encodes to \ + up to {} bytes, over the {}-byte member key cap: shorten or drop a \ + component", + index_name, name, terminal_max_width, MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH, + ))); + } + + // The flat level is itself a GroveDB key. Bounding the encoded + // values above does not bound the concatenated component names. + if let Some(flat_key) = index.flat_level_key() { + if flat_key.len() > usize::from(MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH) { + return Err(structure_error(format!( + "the flat level of index \"{}\" on indexOnly document type \"{}\" \ + encodes to {} bytes, over the {}-byte flat level key cap: shorten \ + or drop a terminal component name", + index_name, + name, + flat_key.len(), + MAX_INDEXED_BYTE_ARRAY_PROPERTY_LENGTH, + ))); } } @@ -2651,7 +2867,7 @@ pub(super) fn apply_index_only( // row — and remove an entry it never created; binding every entry // to its owner closes that, at the cost of the (unneeded) global- // uniqueness-without-owner shape. - if terminal != OWNER_ID + if !index.terminal_contains(OWNER_ID) && !index .properties .iter() @@ -2670,7 +2886,7 @@ pub(super) fn apply_index_only( // `created_at` only when `$createdAt` is in `required`. Without // this, an indexed `$createdAt` would silently take the missing- // value branch instead of storing block time. - if (terminal == CREATED_AT + if (index.terminal_contains(CREATED_AT) || index .properties .iter() @@ -2746,7 +2962,7 @@ pub(super) fn apply_index_only( // index qualifies as the proof index.) let has_proof_index = document_type.indices.values().any(|index| { !index.skip_if_absent - && index.terminal.as_deref() != Some(CREATED_AT) + && !index.terminal_contains(CREATED_AT) && !index .properties .iter() @@ -2787,10 +3003,15 @@ pub(super) fn apply_index_only( continue; } let is_trigger = skip_triggers.contains(property_name.as_str()); + if document_type.entry_payload.contains(property_name.as_str()) { + // Stored in every entry's value: validated above (required, + // bounded, in no index). + continue; + } let covered = document_type.indices.values().any(|index| { // A skip index only counts as coverage for its own trigger. (is_trigger || !index.skip_if_absent) - && (index.terminal.as_deref() == Some(property_name.as_str()) + && (index.terminal_contains(property_name) || index .properties .iter() @@ -2824,7 +3045,7 @@ pub(super) fn apply_index_only( // when no non-skip index (and no deeper level of any index) // reaches through that branch. for (index_name, index) in document_type.indices.iter() { - if index.terminal.as_deref() == Some(property_name.as_str()) { + if index.terminal_contains(property_name) { return Err(structure_error(format!( "optional property \"{}\" on indexOnly document type \"{}\" is the \ terminal of index \"{}\": a terminal is every entry's member key \ diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/index_only_tests.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/index_only_tests.rs index e5cfcd0473d..f7a7e302652 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/index_only_tests.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/index_only_tests.rs @@ -20,6 +20,8 @@ //! validation to pin the meta-schema admission. use super::*; +use crate::consensus::basic::BasicError; +use crate::consensus::ConsensusError; use crate::data_contract::document_type::accessors::DocumentTypeV2Getters; use crate::data_contract::errors::DataContractError; use platform_value::platform_value; @@ -169,6 +171,41 @@ fn expect_structure_error(result: Result, needle: } } +/// The terminal shares the prefix positions' shape checks, which report +/// through the typed index consensus errors rather than a structure error. +fn expect_basic_error( + result: Result, + what: &str, + check: impl Fn(&BasicError) -> bool, +) { + match result { + Err(ProtocolError::ConsensusError(err)) => match *err { + ConsensusError::BasicError(ref basic) if check(basic) => {} + other => panic!("expected {what}, got {other}"), + }, + Err(other) => panic!("expected {what}, got {other}"), + Ok(_) => panic!("expected {what}, but the schema parsed"), + } +} + +/// Add a schema property to the likes schema and list it in `required`. +fn with_required_property(schema: &mut Value, name: &str, definition: Value) { + schema + .get_mut("properties") + .expect("properties accessible") + .expect("properties present") + .set_value(name, definition) + .expect("property applies"); + let mut required = schema + .get_optional_array("required") + .expect("required readable") + .unwrap_or_default(); + required.push(Value::Text(name.to_string())); + schema + .set_value("required", Value::Array(required)) + .expect("required applies"); +} + // ── the happy path ────────────────────────────────────────────────────── #[test] @@ -195,15 +232,15 @@ fn terminal_defaults_to_owner_id() { // `byPost` omits its terminal; normalization spells it out, so the // omitted and explicit forms parse to equal indexes. assert_eq!( - document_type.indices["byPost"].terminal.as_deref(), + document_type.indices["byPost"].single_terminal(), Some("$ownerId") ); assert_eq!( - document_type.indices["byHashtagPost"].terminal.as_deref(), + document_type.indices["byHashtagPost"].single_terminal(), Some("$ownerId") ); assert_eq!( - document_type.indices["byLiker"].terminal.as_deref(), + document_type.indices["byLiker"].single_terminal(), Some("postId") ); } @@ -586,13 +623,155 @@ fn rejects_terminal_repeating_an_index_property() { } #[test] -fn rejects_terminal_without_refers_to() { - // `hashtag` is a plain string — not `$ownerId`, not a refersTo-typed - // identifier — so it cannot be a member key. +fn accepts_a_plain_scalar_terminal() { + // `hashtag` is a bounded string with no refersTo: any property a prefix + // position admits may be the member key, so byLiker becomes + // `[$ownerId] → hashtag` — one entry per (owner, hashtag). let schema = likes_schema_with_index_key(2, "terminal", platform_value!("hashtag")); + for full_validation in [false, true] { + let document_type = parse_with(schema.clone(), PlatformVersion::latest(), full_validation) + .expect("a string terminal parses"); + assert_eq!( + document_type.indices["byLiker"].single_terminal(), + Some("hashtag") + ); + } +} + +#[test] +fn accepts_byte_array_and_integer_terminals() { + // A 33-byte array (a compressed public key) and a bounded integer as + // member keys, each carried by a fourth index so every property stays + // indexed: byLiker becomes `[$ownerId] → pubKey`, byScore is + // `[$ownerId, pubKey] → score`. + let mut schema = likes_schema_with_index_key(2, "terminal", platform_value!("pubKey")); + with_required_property( + &mut schema, + "pubKey", + platform_value!({ + "type": "array", + "byteArray": true, + "minItems": 33, + "maxItems": 33, + "position": 2 + }), + ); + with_required_property( + &mut schema, + "score", + platform_value!({ "type": "integer", "minimum": 0, "maximum": 100, "position": 3 }), + ); + schema + .get_mut("indices") + .expect("indices accessible") + .expect("indices present") + .as_array_mut() + .expect("indices is an array") + .push(platform_value!({ + "name": "byScore", + "properties": [{ "$ownerId": "asc" }, { "pubKey": "asc" }], + "terminal": "score" + })); + for full_validation in [false, true] { + let document_type = parse_with(schema.clone(), PlatformVersion::latest(), full_validation) + .expect("byte array and integer terminals parse"); + assert_eq!( + document_type.indices["byLiker"].single_terminal(), + Some("pubKey") + ); + assert_eq!( + document_type.indices["byScore"].single_terminal(), + Some("score") + ); + } +} + +#[test] +fn object_terminals_are_not_addressable_but_their_leaves_are() { + // Only leaf properties exist in the flattened property map, so an + // object cannot be named as a terminal at all — the same as at a prefix + // position — while a nested leaf can, provided its ancestor is required + // (the no-null invariant every indexOnly property carries). + let mut schema = likes_schema_with_index_key(2, "terminal", platform_value!("profile")); + with_required_property( + &mut schema, + "profile", + platform_value!({ + "type": "object", + "properties": { "nick": { "type": "string", "maxLength": 8, "position": 0 } }, + "required": ["nick"], + "position": 2 + }), + ); + expect_structure_error( + parse_with(schema.clone(), PlatformVersion::latest(), false), + "does not name a property", + ); + + schema + .get_mut("indices") + .expect("indices accessible") + .expect("indices present") + .as_array_mut() + .expect("indices is an array") + .get_mut(2) + .expect("byLiker exists") + .set_value("terminal", platform_value!("profile.nick")) + .expect("terminal applies"); + let document_type = parse_with(schema, PlatformVersion::latest(), false) + .expect("a nested leaf terminal parses"); + assert_eq!( + document_type.indices["byLiker"].single_terminal(), + Some("profile.nick") + ); +} + +#[test] +fn rejects_an_unbounded_byte_array_terminal() { + // 256 bytes exceeds the indexed byte-array bound (255, grovedb's key cap). + let mut schema = likes_schema_with_index_key(2, "terminal", platform_value!("blob")); + with_required_property( + &mut schema, + "blob", + platform_value!({ + "type": "array", + "byteArray": true, + "minItems": 0, + "maxItems": 256, + "position": 2 + }), + ); + expect_basic_error( + parse_with(schema, PlatformVersion::latest(), false), + "InvalidIndexedPropertyConstraintError", + |error| matches!(error, BasicError::InvalidIndexedPropertyConstraintError(_)), + ); +} + +#[test] +fn rejects_an_overlong_string_terminal() { + // 64 characters exceeds the indexed string bound (63). + let mut schema = likes_schema_with_index_key(2, "terminal", platform_value!("caption")); + with_required_property( + &mut schema, + "caption", + platform_value!({ "type": "string", "maxLength": 64, "position": 2 }), + ); + expect_basic_error( + parse_with(schema, PlatformVersion::latest(), false), + "InvalidIndexedPropertyConstraintError", + |error| matches!(error, BasicError::InvalidIndexedPropertyConstraintError(_)), + ); +} + +#[test] +fn rejects_system_property_terminals_other_than_owner_id() { + // `$createdAt` may be indexed in the prefix, where the rules that reason + // about it look; it is not admitted as a terminal. + let schema = likes_schema_with_index_key(2, "terminal", platform_value!("$createdAt")); expect_structure_error( parse_with(schema, PlatformVersion::latest(), false), - "refersTo", + "only $ownerId may be a terminal", ); } @@ -751,46 +930,49 @@ fn rejects_indexed_created_at_that_is_not_required() { } #[test] -fn rejects_identity_public_key_reference_terminals() { - // identityPublicKey is a compound reference (identity id here, key id - // in a companion property) — the member key alone cannot identify the - // referenced key, so it is not a legal terminal. +fn accepts_an_identity_public_key_reference_terminal() { + // An identityPublicKey reference is a 32-byte identifier like any other + // as a member key; the companion key id is indexed in the prefix, so + // entries for different keys of one identity stay distinct. The old + // "referable entity" restriction no longer applies: a terminal is any + // indexable property. let mut schema = likes_schema_with_index_key(2, "terminal", platform_value!("keyRef")); + with_required_property( + &mut schema, + "keyRef", + platform_value!({ + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "refersTo": { "type": "identityPublicKey", "keyIdProperty": "keyId" }, + "position": 2 + }), + ); + with_required_property( + &mut schema, + "keyId", + platform_value!({ "type": "integer", "minimum": 0, "maximum": 100, "position": 3 }), + ); schema - .get_mut("properties") - .expect("properties accessible") - .expect("properties present") - .set_value( - "keyRef", - platform_value!({ - "type": "array", - "byteArray": true, - "minItems": 32, - "maxItems": 32, - "contentMediaType": "application/x.dash.dpp.identifier", - "refersTo": { "type": "identityPublicKey", "keyIdProperty": "keyId" }, - "position": 2 - }), - ) - .expect("property applies"); - schema - .get_mut("properties") - .expect("properties accessible") - .expect("properties present") - .set_value( - "keyId", - platform_value!({ "type": "integer", "minimum": 0, "maximum": 100, "position": 3 }), - ) - .expect("property applies"); - schema + .get_mut("indices") + .expect("indices accessible") + .expect("indices present") + .as_array_mut() + .expect("indices is an array") + .get_mut(2) + .expect("byLiker exists") .set_value( - "required", - platform_value!(["hashtag", "postId", "keyRef", "keyId"]), + "properties", + platform_value!([{ "$ownerId": "asc" }, { "keyId": "asc" }]), ) - .expect("required applies"); - expect_structure_error( - parse_with(schema, PlatformVersion::latest(), false), - "identityPublicKey", + .expect("properties apply"); + let document_type = parse_with(schema, PlatformVersion::latest(), false) + .expect("an identityPublicKey reference terminal parses"); + assert_eq!( + document_type.indices["byLiker"].single_terminal(), + Some("keyRef") ); } @@ -865,6 +1047,458 @@ fn rejects_missing_owner_id() { ); } +// ── composite and flat terminals ──────────────────────────────────────── + +/// The likes schema with one extra index appended. +fn likes_schema_with_extra_index(index: Value) -> Value { + let mut schema = likes_schema(); + schema + .get_mut("indices") + .expect("indices accessible") + .expect("indices present") + .as_array_mut() + .expect("indices is an array") + .push(index); + schema +} + +#[test] +fn accepts_a_composite_terminal_below_a_prefix() { + // byLiker becomes `[$ownerId] → postId ‖ hashtag`: a 32-byte identifier + // followed by a string, which may only be the last component. + let mut schema = + likes_schema_with_index_key(2, "terminal", platform_value!(["postId", "hashtag"])); + // The hashtag is bounded to 40 characters here: a string bound counts + // characters of up to four bytes, and 32 + 4 × 63 would exceed the + // 255-byte member key cap. + schema + .get_mut("properties") + .expect("properties accessible") + .expect("properties present") + .set_value( + "hashtag", + platform_value!({ "type": "string", "maxLength": 40, "position": 0 }), + ) + .expect("property applies"); + for full_validation in [false, true] { + let document_type = parse_with(schema.clone(), PlatformVersion::latest(), full_validation) + .expect("a composite terminal parses"); + let index = &document_type.indices["byLiker"]; + assert_eq!( + index.terminal_components(), + &["postId".to_string(), "hashtag".to_string()] + ); + assert!(index.single_terminal().is_none()); + assert!(!index.is_flat()); + } +} + +#[test] +fn accepts_a_flat_composite_terminal() { + // An index with no properties at all: a flat index keyed by + // `postId ‖ $ownerId`, living under its own zero-byte-prefixed level. + let schema = likes_schema_with_extra_index(platform_value!({ + "name": "byPostOwner", + "terminal": ["postId", "$ownerId"] + })); + for full_validation in [false, true] { + let document_type = parse_with(schema.clone(), PlatformVersion::latest(), full_validation) + .expect("a flat composite terminal parses"); + let index = &document_type.indices["byPostOwner"]; + assert!(index.is_flat()); + assert!(index.properties.is_empty()); + assert_eq!( + index.flat_level_key().as_deref(), + Some("\0postId\0$ownerId") + ); + assert!( + document_type + .index_structure + .sub_levels() + .contains_key("\0postId\0$ownerId"), + "the flat level is a top-level entry of the index structure" + ); + assert!( + document_type.index_structure.sub_levels()["\0postId\0$ownerId"] + .has_index_with_type() + .is_some(), + "the flat level terminates its index" + ); + } +} + +#[test] +fn rejects_a_variable_width_leading_component() { + // A string is never fixed width, so it cannot be followed by another + // component. + let schema = likes_schema_with_index_key(2, "terminal", platform_value!(["hashtag", "postId"])); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "is not fixed width", + ); +} + +#[test] +fn should_bound_flat_level_names_independently_of_member_values() { + // Four booleans and an owner encode to only 36 member-key bytes, but + // the names used for their flat level can exceed GroveDB's key limit. + for last_name_length in [50, 51, 64] { + let names = [ + "a".repeat(64), + "b".repeat(64), + "c".repeat(64), + "d".repeat(last_name_length), + ]; + let mut schema = platform_value!({ + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "properties": {}, + "required": [], + "additionalProperties": false + }); + for (position, name) in names.iter().enumerate() { + with_required_property( + &mut schema, + name, + platform_value!({ "type": "boolean", "position": position }), + ); + } + let mut terminal: Vec = names.into_iter().map(Value::Text).collect(); + terminal.push(Value::Text("$ownerId".to_string())); + schema + .set_value( + "indices", + platform_value!([{ + "name": "byValues", + "terminal": terminal + }]), + ) + .expect("indices apply"); + + for full_validation in [false, true] { + let result = parse_with(schema.clone(), PlatformVersion::latest(), full_validation); + if last_name_length == 50 { + let document_type = result.expect("a 255-byte flat level key is accepted"); + assert_eq!( + document_type.indices["byValues"] + .flat_level_key() + .unwrap() + .len(), + 255 + ); + } else { + expect_structure_error(result, "flat level key cap"); + } + } + } +} + +#[test] +fn rejects_a_composite_terminal_over_the_key_cap() { + // Two 200-byte arrays: 400 bytes, over grovedb's 255-byte key cap. + let mut schema = + likes_schema_with_index_key(2, "terminal", platform_value!(["blobA", "blobB"])); + for (name, position) in [("blobA", 2), ("blobB", 3)] { + with_required_property( + &mut schema, + name, + platform_value!({ + "type": "array", + "byteArray": true, + "minItems": 200, + "maxItems": 200, + "position": position + }), + ); + } + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "member key cap", + ); +} + +#[test] +fn rejects_a_terminal_component_name_with_a_zero_byte() { + let mut schema = login_response_schema(); + with_required_property( + &mut schema, + "bad\0name", + platform_value!({ + "type": "array", + "byteArray": true, + "minItems": 4, + "maxItems": 4, + "position": 3 + }), + ); + schema + .set_value( + "indices", + platform_value!([ + { "name": "byRequest", "terminal": ["bad\0name", "$ownerId"] } + ]), + ) + .expect("indices apply"); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "zero byte", + ); +} + +#[test] +fn rejects_a_terminal_with_no_components() { + let schema = likes_schema_with_index_key(2, "terminal", platform_value!([])); + match parse_with(schema, PlatformVersion::latest(), false) { + Ok(_) => panic!("an empty terminal list must be refused"), + Err(error) => assert!( + error.to_string().contains("at least one property"), + "expected the empty-terminal rejection, got: {error}" + ), + } +} + +/// The flat level's storage key is the zero-joined component names, which +/// the value-width cap does not bound: five one-byte components named at +/// the 64-character limit fit the member key with room to spare and still +/// spell a 300-byte level key. +#[test] +fn rejects_a_flat_level_key_over_the_key_cap() { + let names: Vec = (0..5) + .map(|position| format!("{position}{}", "n".repeat(63))) + .collect(); + let mut properties = platform_value::Value::Map(Default::default()); + for (position, name) in names.iter().enumerate() { + properties + .set_value( + name, + platform_value!({ + "type": "array", + "byteArray": true, + "minItems": 1, + "maxItems": 1, + "position": position as u64 + }), + ) + .expect("property applies"); + } + let required: Vec = names + .iter() + .map(|name| platform_value!(name.as_str())) + .collect(); + let mut terminal = required.clone(); + terminal.push(platform_value!("$ownerId")); + let schema = platform_value!({ + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [{ "name": "byNames", "terminal": platform_value::Value::Array(terminal) }], + "properties": properties, + "required": platform_value::Value::Array(required), + "additionalProperties": false + }); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "flat level key cap", + ); +} + +#[test] +fn rejects_a_terminal_naming_a_component_twice() { + let schema = likes_schema_with_index_key(2, "terminal", platform_value!(["postId", "postId"])); + match parse_with(schema, PlatformVersion::latest(), false) { + Ok(_) => panic!("a duplicate terminal component must be refused"), + Err(error) => assert!( + error.to_string().contains("twice"), + "expected the duplicate-component refusal, got {error}" + ), + } +} + +#[test] +fn rejects_aggregates_on_a_flat_index() { + // No prefix level exists for an aggregate to apply to. + let schema = likes_schema_with_extra_index(platform_value!({ + "name": "byPostOwner", + "terminal": ["postId", "$ownerId"], + "countable": true + })); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "has no properties (a flat index", + ); +} + +#[test] +fn rejects_a_flat_index_on_a_stored_type() { + let schema = platform_value!({ + "type": "object", + "properties": { + "hashtag": { "type": "string", "maxLength": 63, "position": 0 } + }, + "indices": [{ "name": "flat" }], + "additionalProperties": false + }); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "only allowed on an indexOnly document type", + ); +} + +// ── entry payload ─────────────────────────────────────────────────────── + +/// The app-connect login response: a flat index keyed by the request hash +/// and the responding identity, with the wallet's ephemeral key and the +/// ciphertext in every entry's value. +fn login_response_schema() -> Value { + platform_value!({ + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] } + ], + "entryPayload": ["walletEphemeralPubKey", "encryptedPayload"], + "properties": { + "appEphemeralPubKeyHash": { + "type": "array", + "byteArray": true, + "minItems": 20, + "maxItems": 20, + "position": 0 + }, + "walletEphemeralPubKey": { + "type": "array", + "byteArray": true, + "minItems": 33, + "maxItems": 33, + "position": 1 + }, + "encryptedPayload": { + "type": "array", + "byteArray": true, + "minItems": 60, + "maxItems": 572, + "position": 2 + } + }, + "required": ["appEphemeralPubKeyHash", "walletEphemeralPubKey", "encryptedPayload"], + "additionalProperties": false + }) +} + +#[test] +fn accepts_an_entry_payload_on_a_flat_composite_index() { + for full_validation in [false, true] { + let document_type = parse_with( + login_response_schema(), + PlatformVersion::latest(), + full_validation, + ) + .expect("the login response schema parses"); + assert_eq!( + document_type + .entry_payload + .iter() + .map(String::as_str) + .collect::>(), + vec!["encryptedPayload", "walletEphemeralPubKey"], + "payload properties are kept in name order" + ); + assert!(document_type.indices["byRequest"].is_flat()); + } +} + +#[test] +fn rejects_an_entry_payload_property_that_is_also_indexed() { + let mut schema = login_response_schema(); + schema + .set_value( + "entryPayload", + platform_value!([ + "appEphemeralPubKeyHash", + "walletEphemeralPubKey", + "encryptedPayload" + ]), + ) + .expect("entryPayload applies"); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "also appears in index", + ); +} + +#[test] +fn rejects_an_unbounded_entry_payload_property() { + let mut schema = login_response_schema(); + schema + .get_mut("properties") + .expect("properties accessible") + .expect("properties present") + .set_value( + "encryptedPayload", + platform_value!({ "type": "array", "byteArray": true, "position": 2 }), + ) + .expect("property applies"); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "must be bounded", + ); +} + +#[test] +fn rejects_an_optional_entry_payload_property() { + let mut schema = login_response_schema(); + schema + .set_value( + "required", + platform_value!(["appEphemeralPubKeyHash", "walletEphemeralPubKey"]), + ) + .expect("required applies"); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "must be listed in `required`", + ); +} + +#[test] +fn rejects_an_entry_payload_naming_an_unknown_property() { + let mut schema = login_response_schema(); + schema + .set_value( + "entryPayload", + platform_value!(["nonsense", "encryptedPayload"]), + ) + .expect("entryPayload applies"); + // walletEphemeralPubKey is then in no index and no payload either, but + // the unknown name is refused first. + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "is not a top-level property", + ); +} + +#[test] +fn rejects_an_entry_payload_on_a_stored_type() { + let schema = platform_value!({ + "type": "object", + "properties": { + "hashtag": { "type": "string", "maxLength": 63, "position": 0 }, + "note": { "type": "string", "maxLength": 63, "position": 1 } + }, + "required": ["hashtag", "note"], + "indices": [{ "name": "byHashtag", "properties": [{ "hashtag": "asc" }] }], + "entryPayload": ["note"], + "additionalProperties": false + }); + expect_structure_error( + parse_with(schema, PlatformVersion::latest(), false), + "declares `entryPayload`", + ); +} + // ── terminal without indexOnly ────────────────────────────────────────── #[test] @@ -1135,8 +1769,7 @@ fn accepts_a_deletable_document_reference_as_the_terminal() { .indices .get("byLiker") .unwrap() - .terminal - .as_deref(), + .single_terminal(), Some("postId") ); } diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs index 89833f017cb..f1478ec756c 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs @@ -287,6 +287,8 @@ fn try_from_schema_generation_3( // above. let aggregates = common::parse_doctype_aggregate_keywords(&schema, name)?; let index_only = common::parse_index_only_keyword(&schema)?; + let entry_payload = + common::parse_property_name_list_keyword(&schema, name, property_names::ENTRY_PAYLOAD)?; let action_fees = DocumentActionFees::try_from_document_schema(&schema, name)?; let can_be_deleted_by_moderators = common::parse_can_be_deleted_by_moderators_keyword(&schema)?; let can_be_deleted_by_moderators_for = @@ -362,11 +364,12 @@ fn try_from_schema_generation_3( let mut v2: DocumentTypeV2 = v1.into(); v2.action_fees = action_fees; + v2.entry_payload = entry_payload; common::apply_doctype_aggregates(&mut v2, aggregates, name)?; // After the aggregates: `apply_index_only` rejects the doctype-level // aggregate flags (they describe the primary-key tree, which an // indexOnly type does not have), so it has to see them already applied. - common::apply_index_only(&mut v2, index_only, name)?; + common::apply_index_only(&mut v2, index_only, name, platform_version)?; // After the core parse: the lints read the resolved `documentsMutable` // flag (contract default applied) and the parsed top-level properties. common::apply_immutable_fields( diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/moderators_delete_tests.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/moderators_delete_tests.rs index e019653d73d..ded42ce7679 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/moderators_delete_tests.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/moderators_delete_tests.rs @@ -1,6 +1,8 @@ //! The `canBeDeletedByModerators` doctype keyword (protocol version 14): what it requires of //! the contract and of the document type. use super::*; +use crate::consensus::basic::BasicError; +use crate::consensus::ConsensusError; use crate::data_contract::config::moderation::{ContractModerationConfig, ContractModerators}; use crate::data_contract::document_type::accessors::DocumentTypeV2Getters; use platform_value::platform_value; @@ -84,6 +86,40 @@ fn assert_refused_naming(result: Result, fragments: } } +#[test] +fn should_preserve_contract_structure_errors_for_non_object_schemas() { + let platform_version = PlatformVersion::latest(); + let config = moderated_config(platform_version); + for schema in [ + Value::Null, + Value::Bool(false), + Value::Array(vec![]), + Value::Text("invalid".to_string()), + Value::U64(1), + ] { + for full_validation in [false, true] { + let error = parse_with_config( + schema.clone(), + &config, + platform_version.protocol_version, + full_validation, + ) + .expect_err("a document schema must be an object"); + let ProtocolError::ConsensusError(error) = error else { + panic!("expected a consensus error, got {error:?}"); + }; + assert!( + matches!(error.as_ref(), + ConsensusError::BasicError(BasicError::ContractError( + DataContractError::InvalidContractStructure(message) + )) if message == "document schema must be an object: structure error: value is not a map" + ), + "schema {schema:?}, full_validation={full_validation}: {error:?}", + ); + } + } +} + #[test] fn should_parse_the_flag_on_a_moderated_contract() { let document_type = parse_moderated(post_schema(platform_value!({}))).expect("parse"); diff --git a/packages/rs-dpp/src/data_contract/document_type/index/mod.rs b/packages/rs-dpp/src/data_contract/document_type/index/mod.rs index 888d1648d64..b3f44dbc7d3 100644 --- a/packages/rs-dpp/src/data_contract/document_type/index/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/index/mod.rs @@ -71,12 +71,16 @@ pub const TIME_RANGE: &str = "timeRange"; /// `range % step == 0`), not a versioned limit: it is part of what makes a /// transform well-formed at all. pub const MAX_TIME_RANGE_PHASE_SECONDS: u64 = 31_536_000; -/// Index-level keyword naming the property whose value is the index entry's -/// **member key** on an `indexOnly` document type — the docId-analog terminal -/// key stored under the `0` storage marker, where a normal index stores the -/// document id. `"$ownerId"` (the default) or a refersTo-typed identifier -/// property (identity, contract, token, or permanent document — the permanent -/// kinds `refersTo` targets). Only allowed on indexOnly document types; the +/// Index-level keyword naming the property — or the ordered list of +/// properties, for a composite terminal — whose encoded value(s), +/// concatenated, are the index entry's **member key** on an `indexOnly` +/// document type: the docId-analog terminal key stored under the `0` storage +/// marker, where a normal index stores the document id. Each component is +/// `"$ownerId"` (the default) or any schema property a prefix position could +/// carry (identifiers with or without a `refersTo`, bounded byte arrays and +/// strings, integers, booleans, dates), every component but the last fixed +/// width. An index with no `properties` is a flat index keyed by its +/// terminal alone. Only allowed on indexOnly document types; the /// doc-type-level validation rejects it elsewhere. Meta-schema v3+ (protocol /// version 14). pub const TERMINAL: &str = "terminal"; @@ -429,6 +433,70 @@ where ) } +/// Deserializer for [`Index::terminal`] accepting `null`, a bare property +/// name (the single-component form, and the only spelling the field had +/// while it was an `Option`) and an array of names (the composite +/// form). Serialization always emits the array form. +#[cfg(feature = "serde-conversion")] +fn deserialize_terminal<'de, D>(deserializer: D) -> Result>, D::Error> +where + D: serde::Deserializer<'de>, +{ + #[derive(Deserialize)] + #[serde(untagged)] + enum TerminalCompat { + Many(Vec), + One(String), + } + Ok(match Option::::deserialize(deserializer)? { + None => None, + Some(TerminalCompat::One(name)) => Some(vec![name]), + Some(TerminalCompat::Many(names)) => Some(names), + }) +} + +/// Whether `fields`, all terminal components, appear in the terminal's +/// declared order: one member key sorts by its components in that order, +/// so a walk over member keys can implement no other ordering. The terminal +/// route additionally requires the run to be contiguous after the +/// equality-bound components; the matcher only refuses what no index walk +/// could serve, so another index may still take the query. +fn follows_component_order(components: &[String], fields: &[&str]) -> bool { + let position_of = |field: &str| components.iter().position(|component| component == field); + fields + .windows(2) + .all(|pair| match (position_of(pair[0]), position_of(pair[1])) { + (Some(earlier), Some(later)) => earlier < later, + _ => false, + }) +} + +/// The storage key of a flat indexOnly index's level (an index with no +/// prefix `properties`): a zero byte followed by each terminal component +/// name, every name preceded by a zero byte. Property names never contain a +/// zero byte, so a flat level can never collide with a property-name tree: +/// a flat `["$ownerId"]` terminal keys `"\0$ownerId"`, beside the +/// `"$ownerId"` property-name tree a prefixed index may own in the same +/// document type. Every place that turns a flat index into a GroveDB path +/// segment — contract setup, the document walkers (via `IndexLevel`), the +/// uniqueness and delete probes, query path derivation, synthesis and proof +/// verification — derives it through this function. +pub fn flat_level_key_for(components: &[String]) -> String { + let mut key = String::with_capacity(components.iter().map(|c| c.len() + 1).sum()); + for component in components { + key.push('\0'); + key.push_str(component); + } + key +} + +/// Whether an index-structure level key names a flat index's level (see +/// [`flat_level_key_for`]) rather than a property-name or grid-qualified +/// tree. +pub fn is_flat_level_key(level_key: &str) -> bool { + level_key.starts_with('\0') +} + // Indices documentation: https://dashplatform.readme.io/docs/reference-data-contracts#document-indices #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] #[cfg_attr(feature = "serde-conversion", derive(Serialize, Deserialize))] @@ -625,21 +693,31 @@ pub struct Index { // JSON must still deserialize. #[cfg_attr(feature = "serde-conversion", serde(default))] pub time_range: Option, - /// On an `indexOnly` document type, the property whose value is this - /// index's member key: the terminal key under the `0` storage marker, - /// sitting exactly where a normal index stores the document id — except - /// the element is an `Item` instead of a `Reference`, because there is no - /// primary-storage row to reference. `"$ownerId"` or a refersTo-typed - /// identifier property; the doc-type-level validation - /// (`apply_index_only`) normalizes an omitted value to `"$ownerId"` and + /// On an `indexOnly` document type, the property — or the ordered list + /// of properties, for a composite terminal — whose encoded value(s), + /// concatenated, form this index's member key: the terminal key under + /// the `0` storage marker, sitting exactly where a normal index stores + /// the document id — except the element is an `Item` instead of a + /// `Reference`, because there is no primary-storage row to reference. + /// Each component is `"$ownerId"` or any schema property a prefix + /// position could carry, keyed by its tree-key encoding; every component + /// but the last must be fixed width so equality on the leading ones is a + /// clean key range. An index with no `properties` at all is a *flat* + /// index: its entries live directly under a level keyed by the terminal's + /// names ([`flat_level_key_for`]). The doc-type-level validation + /// (`apply_index_only`) normalizes an omitted value to `["$ownerId"]` and /// rejects the keyword entirely on non-indexOnly document types, so on a /// parsed non-indexOnly type this is always `None`. // // `serde(default)`: added after the struct's serde shape was in the wild // (see the note on `countable` above), so pre-existing JSON must still - // deserialize. - #[cfg_attr(feature = "serde-conversion", serde(default))] - pub terminal: Option, + // deserialize. The deserializer also accepts the bare-string spelling + // the field had while it was an `Option`. + #[cfg_attr( + feature = "serde-conversion", + serde(default, deserialize_with = "deserialize_terminal") + )] + pub terminal: Option>, /// On an indexOnly document type whose index path is fully determined by /// a same-contract `permanentDocument` reference (see [`PREALLOCATED`]): /// when `true`, inserting a referenced document also creates this index's @@ -801,6 +879,43 @@ impl Index { } } + /// The terminal's components: one name for a single-property terminal, + /// several for a composite one (the member key is their encoded values + /// concatenated in this order), none on a non-indexOnly index. + pub fn terminal_components(&self) -> &[String] { + self.terminal.as_deref().unwrap_or(&[]) + } + + /// Whether `name` is one of the terminal's components. + pub fn terminal_contains(&self, name: &str) -> bool { + self.terminal_components() + .iter() + .any(|component| component == name) + } + + /// The terminal's single component; `None` on a composite terminal or a + /// non-indexOnly index. + pub fn single_terminal(&self) -> Option<&str> { + match self.terminal.as_deref() { + Some([component]) => Some(component.as_str()), + _ => None, + } + } + + /// Whether this is a flat index: an indexOnly index with no prefix + /// properties, whose entries live directly under the level keyed by + /// [`Self::flat_level_key`]. + pub fn is_flat(&self) -> bool { + self.properties.is_empty() && self.terminal.is_some() + } + + /// The storage key of a flat index's level (see [`flat_level_key_for`]); + /// `None` unless [`Self::is_flat`]. + pub fn flat_level_key(&self) -> Option { + self.is_flat() + .then(|| flat_level_key_for(self.terminal_components())) + } + /// Get values pub fn extract_values(&self, data: &BTreeMap) -> Vec { self.properties @@ -905,29 +1020,36 @@ impl Index { in_field_name: Option<&str>, order_by: &[&str], ) -> Option<(u16, bool)> { - let Some(terminal) = self.terminal.as_deref() else { + let Some(components) = self.terminal.as_deref() else { return self .matches(index_names, in_field_name, order_by) .map(|difference| (difference, false)); }; + let is_component = |field: &str| components.iter().any(|component| component == field); - let terminal_used = index_names.contains(&terminal); + let terminal_used = index_names.iter().any(|field| is_component(field)); let prefix_fields: Vec<&str> = index_names .iter() .copied() - .filter(|field| *field != terminal) + .filter(|field| !is_component(field)) .collect(); - let prefix_order_by: &[&str] = match order_by.iter().position(|field| *field == terminal) { - // Ordering by the terminal is ordering the deepest level — - // admissible only as the ordering's last entry. - Some(position) if position + 1 == order_by.len() => &order_by[..position], + let prefix_order_by: &[&str] = match order_by.iter().position(|field| is_component(field)) { + // Ordering by the terminal (any of its components) is ordering + // the deepest level — admissible only as the ordering's + // trailing entries. + Some(position) + if order_by[position..].iter().all(|field| is_component(field)) + && follows_component_order(components, &order_by[position..]) => + { + &order_by[..position] + } Some(_) => return None, None => order_by, }; let prefix_in_field = match in_field_name { // An `in` on the terminal sits at the deepest position by // construction; the prefix keeps no `in` constraint. - Some(field) if field == terminal => None, + Some(field) if is_component(field) => None, other => other, }; @@ -946,6 +1068,14 @@ impl Index { in_field_name: Option<&str>, order_by: &[&str], ) -> Option { + // A FLAT indexOnly index has no prefix components at all: it + // matches exactly the queries that bind none (its terminal + // components are matched by the terminal-aware callers, which + // strip them before reaching here). + if properties.is_empty() { + return (index_names.is_empty() && in_field_name.is_none() && order_by.is_empty()) + .then_some(0); + } // Here we are trying to figure out if the Index matches the order by // To do so we take the index and go backwards as we need the order by clauses to be // continuous, but they do not need to be at the end. @@ -1053,27 +1183,34 @@ impl Index { in_field_name: Option<&str>, order_by: &[&str], ) -> Option<(u16, bool)> { - let Some(terminal) = self.terminal.as_deref() else { + let Some(components) = self.terminal.as_deref() else { return self .matches_contiguous(equality_fields, range_field, in_field_name, order_by) .map(|difference| (difference, false)); }; + let is_component = |field: &str| components.iter().any(|component| component == field); - let terminal_used = equality_fields.contains(&terminal) - || range_field == Some(terminal) - || in_field_name == Some(terminal) - || order_by.contains(&terminal); + let terminal_used = equality_fields.iter().any(|field| is_component(field)) + || range_field.is_some_and(is_component) + || in_field_name.is_some_and(is_component) + || order_by.iter().any(|field| is_component(field)); let prefix_equality_fields: Vec<&str> = equality_fields .iter() .copied() - .filter(|field| *field != terminal) + .filter(|field| !is_component(field)) .collect(); - let prefix_range_field = range_field.filter(|field| *field != terminal); - let prefix_in_field = in_field_name.filter(|field| *field != terminal); - let prefix_order_by: &[&str] = match order_by.iter().position(|field| *field == terminal) { - // Ordering by the terminal is ordering the deepest level — - // admissible only as the ordering's last entry. - Some(position) if position + 1 == order_by.len() => &order_by[..position], + let prefix_range_field = range_field.filter(|field| !is_component(field)); + let prefix_in_field = in_field_name.filter(|field| !is_component(field)); + let prefix_order_by: &[&str] = match order_by.iter().position(|field| is_component(field)) { + // Ordering by the terminal (any of its components) is ordering + // the deepest level — admissible only as the ordering's + // trailing entries. + Some(position) + if order_by[position..].iter().all(|field| is_component(field)) + && follows_component_order(components, &order_by[position..]) => + { + &order_by[..position] + } Some(_) => return None, None => order_by, }; @@ -1294,7 +1431,7 @@ impl Index { let mut ranked_summable = false; let mut ranked_averageable = false; let mut time_range: Option = None; - let mut terminal: Option = None; + let mut terminal: Option> = None; let mut preallocated = false; let mut skip_if_absent = false; @@ -1731,19 +1868,51 @@ impl Index { // fact this parser cannot see; `apply_index_only` in // `try_from_schema::common` enforces it. TERMINAL if terminal_allowed => { - let terminal_name = - value_value - .as_text() - .ok_or(DataContractError::ValueWrongType( - "terminal value must be a string naming a property".to_string(), - ))?; - if terminal_name.is_empty() { + // A bare name is a single-component terminal; an array + // is a composite one, keyed by the concatenation of its + // components' encoded values in the listed order. + let components: Vec = match value_value { + Value::Text(terminal_name) => vec![terminal_name.clone()], + Value::Array(entries) => entries + .iter() + .map(|entry| { + entry.as_text().map(str::to_owned).ok_or( + DataContractError::ValueWrongType( + "every terminal component must be a string naming a \ + property" + .to_string(), + ), + ) + }) + .collect::>()?, + _ => { + return Err(DataContractError::ValueWrongType( + "terminal value must be a property name or an array of \ + property names" + .to_string(), + )) + } + }; + if components.is_empty() { + return Err(DataContractError::InvalidContractStructure( + "terminal must name at least one property".to_string(), + )); + } + if components.iter().any(|component| component.is_empty()) { return Err(DataContractError::InvalidContractStructure( "terminal must name a property; an empty string names nothing" .to_string(), )); } - terminal = Some(terminal_name.to_owned()); + for (position, component) in components.iter().enumerate() { + if components[..position].contains(component) { + return Err(DataContractError::InvalidContractStructure(format!( + "terminal lists property \"{}\" twice", + component + ))); + } + } + terminal = Some(components); } // `preallocated` is guarded the same way as `terminal` above: // it joined the grammar at meta-schema v3, so below that the @@ -3740,10 +3909,54 @@ mod tests { ); } + /// A composite terminal's components share one member key, so an + /// ordering over them is admissible only in declared order. + #[test] + fn test_matches_including_terminal_contiguous_composite_order() { + let mut index = make_index("idx", vec![("hashtag", true)], false); + index.terminal = Some(vec!["post".to_string(), "owner".to_string()]); + + assert_eq!( + index.matches_including_terminal_contiguous( + &["hashtag"], + None, + None, + &["post", "owner"] + ), + Some((0, true)), + "declared order is admissible" + ); + assert_eq!( + index.matches_including_terminal_contiguous( + &["hashtag", "post"], + None, + None, + &["owner"] + ), + Some((0, true)), + "a later component alone is admissible" + ); + assert_eq!( + index.matches_including_terminal_contiguous( + &["hashtag"], + None, + None, + &["owner", "post"] + ), + None, + "a reversed run cannot be served by any member-key walk" + ); + assert_eq!( + index.matches_including_terminal(&["hashtag"], None, &["owner", "post"]), + None, + "the non-contiguous matcher refuses the same run" + ); + } + #[test] fn test_matches_including_terminal_contiguous() { let mut index = make_index("idx", vec![("hashtag", true), ("post", true)], false); - index.terminal = Some("owner".to_string()); + index.terminal = Some(vec!["owner".to_string()]); // Fully determined prefix + terminal equality. assert_eq!( diff --git a/packages/rs-dpp/src/data_contract/document_type/index/preallocation.rs b/packages/rs-dpp/src/data_contract/document_type/index/preallocation.rs index d97ca664e64..dccb6e19d8b 100644 --- a/packages/rs-dpp/src/data_contract/document_type/index/preallocation.rs +++ b/packages/rs-dpp/src/data_contract/document_type/index/preallocation.rs @@ -234,7 +234,7 @@ mod tests { ranked_summable: false, ranked_averageable: false, time_range: None, - terminal: Some("$ownerId".to_string()), + terminal: Some(vec!["$ownerId".to_string()]), preallocated: true, skip_if_absent: false, } diff --git a/packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs b/packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs index cac5dca3df3..b24540e0047 100644 --- a/packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs @@ -102,18 +102,19 @@ pub struct IndexLevelTypeInfo { /// other two. The set of axes declared here is what the rs-drive write path /// turns into the indexed tree's axis list. pub ranked_averageable: bool, - /// On an indexOnly document type, the property whose value is this - /// index's member key: the terminal key under the `0` storage marker, - /// where a normal index stores the document id — stored as an `Item` - /// instead of a `Reference` because there is no primary-storage row. - /// Always `Some` here when the declaring type is indexOnly (the parser - /// normalizes an omitted terminal to `$ownerId`), always `None` - /// otherwise. Carried on the level info because index levels merge + /// On an indexOnly document type, the property — or ordered list of + /// properties, for a composite terminal — whose encoded value(s) form + /// this index's member key: the terminal key under the `0` storage + /// marker, where a normal index stores the document id — stored as an + /// `Item` instead of a `Reference` because there is no primary-storage + /// row. Always `Some` here when the declaring type is indexOnly (the + /// parser normalizes an omitted terminal to `["$ownerId"]`), always + /// `None` otherwise. Carried on the level info because index levels merge /// across indexes sharing prefixes, and the write path only sees the /// level at the terminal — but two indexes can never share a full /// property list (duplicates are rejected), so each terminating level /// belongs to exactly one index and the field is unambiguous. - pub terminal: Option, + pub terminal: Option>, /// Whether the terminating index is `preallocated` (see /// [`crate::data_contract::document_type::index::PREALLOCATED`]): its /// dynamic trees are created when a refersTo-referenced document is, and @@ -124,6 +125,15 @@ pub struct IndexLevelTypeInfo { /// `false` on every pre-PV14 contract (the grammar rejects the keyword /// below meta-schema v3). pub preallocated: bool, + /// Whether the terminating index is FLAT (an indexOnly index with no + /// prefix properties, see [`Index::is_flat`]): its entries sit directly + /// under the `0` bucket of its own level, which is registration-time + /// structure like a property-name tree, so the delete walker stops its + /// upward prune at that bucket exactly as on a preallocated index. + /// Carried here so the walkers read the layout off the level info that + /// defines it instead of inferring it from a path height. `false` on + /// every pre-PV14 contract and on every prefixed index. + pub flat: bool, } impl IndexType { @@ -356,6 +366,43 @@ impl IndexLevel { .filter_map(|at| index.properties.iter().position(|p| &p.name == at)) .collect(); let min_ranked_at_position = ranked_at_positions.iter().copied().min(); + // A FLAT indexOnly index has no prefix properties: its entries + // live directly under one level keyed by the terminal's + // component names (`flat_level_key_for`, whose zero-byte + // prefix keeps it disjoint from every property-name tree). A + // property-less index with no terminal cannot exist on a + // parsed type — the parser refuses it — so a hand-built one + // stamps nothing here rather than a level nothing could key. + if index.properties.is_empty() { + let Some(flat_key) = index.flat_level_key() else { + continue; + }; + let flat_level = + index_level + .sub_index_levels + .entry(flat_key) + .or_insert_with(|| { + counter += 1; + IndexLevel { + level_identifier: counter, + sub_index_levels: Default::default(), + has_index_with_type: None, + time_range: None, + ranked_count_grouping: false, + count_propagating: false, + count_exempt_branch: false, + } + }); + if flat_level.has_index_with_type.is_some() { + return Err(ConsensusError::BasicError(BasicError::DuplicateIndexError( + DuplicateIndexError::new(document_type_name.to_owned(), index.name.clone()), + )) + .into()); + } + flat_level.has_index_with_type = Some(Self::terminator_info(index)); + continue; + } + let mut current_level = &mut index_level; let mut properties_iter = index.properties.iter().enumerate().peekable(); @@ -432,41 +479,7 @@ impl IndexLevel { .into()); } - let index_type = if index.unique { - UniqueIndex - } else { - NonUniqueIndex - }; - - // if things are null searchable that means we should insert with all null - - current_level.has_index_with_type = Some(IndexLevelTypeInfo { - should_insert_with_all_null: index.null_searchable, - index_type, - countable: index.countable, - range_countable: index.range_countable, - summable: index.summable.clone(), - range_summable: index.range_summable, - // The ranking axes live on the same terminating level - // as the range axes they extend: this is the level - // named after the index's LAST property, whose children - // are that property's value trees (one per group). The - // rs-drive write path reads them off the very same - // `IndexLevelTypeInfo` it already consults for - // `range_countable` / `range_summable` when it picks - // the property-name tree variant. - ranked_countable: index.ranked_countable, - ranked_summable: index.ranked_summable, - ranked_averageable: index.ranked_averageable, - // indexOnly member key. Only ever `Some` on PV14+ - // contracts (the grammar rejects the keyword below - // generation 3), so stamping it here changes nothing - // for any historical index level. - terminal: index.terminal.clone(), - // Same PV14+ gating as `terminal` — `false` on - // every historical index level. - preallocated: index.preallocated, - }); + current_level.has_index_with_type = Some(Self::terminator_info(index)); } } } @@ -483,6 +496,48 @@ impl IndexLevel { Ok(index_level) } + /// The terminator stamp an index leaves on the level its last property + /// reaches (or, for a flat index, on its flat level): the index type and + /// every per-index axis the write path reads off the level. + fn terminator_info(index: &Index) -> IndexLevelTypeInfo { + let index_type = if index.unique { + UniqueIndex + } else { + NonUniqueIndex + }; + // if things are null searchable that means we should insert with all null + IndexLevelTypeInfo { + should_insert_with_all_null: index.null_searchable, + index_type, + countable: index.countable, + range_countable: index.range_countable, + summable: index.summable.clone(), + range_summable: index.range_summable, + // The ranking axes live on the same terminating level as the + // range axes they extend: this is the level named after the + // index's LAST property, whose children are that property's + // value trees (one per group). The rs-drive write path reads + // them off the very same `IndexLevelTypeInfo` it already + // consults for `range_countable` / `range_summable` when it + // picks the property-name tree variant. + ranked_countable: index.ranked_countable, + ranked_summable: index.ranked_summable, + ranked_averageable: index.ranked_averageable, + // indexOnly member key. Only ever `Some` on PV14+ contracts + // (the grammar rejects the keyword below generation 3), so + // stamping it here changes nothing for any historical index + // level. + terminal: index.terminal.clone(), + // Same PV14+ gating as `terminal` — `false` on every + // historical index level. + preallocated: index.preallocated, + // A flat index terminates on its own level, directly under the + // document type: the one layout whose prune boundary is the + // level's `0` bucket rather than the document type. + flat: index.is_flat(), + } + } + /// Recursively marks, under every prefix-ranking chain level (grouping /// or count-propagating), the child levels that do NOT continue the /// chain as [`Self::count_exempt_branch`]. The chain child is the one diff --git a/packages/rs-dpp/src/data_contract/document_type/methods/validate_update/common/mod.rs b/packages/rs-dpp/src/data_contract/document_type/methods/validate_update/common/mod.rs index 1985bc056ae..790c0a6d7fa 100644 --- a/packages/rs-dpp/src/data_contract/document_type/methods/validate_update/common/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/methods/validate_update/common/mod.rs @@ -390,6 +390,22 @@ impl DocumentTypeRef<'_> { ); } + // The entry payload is part of every stored entry's value layout: + // adding, dropping or renaming a payload property would leave the + // existing entries undecodable (and their commitments unrecomputable). + if new_document_type.entry_payload() != self.entry_payload() { + return SimpleConsensusValidationResult::new_with_error( + DocumentTypeUpdateError::new( + self.data_contract_id(), + self.name(), + "document type can not change its entryPayload: the listed properties are \ + the value layout of every stored entry" + .to_string(), + ) + .into(), + ); + } + SimpleConsensusValidationResult::new() } @@ -1360,6 +1376,118 @@ mod tests { ); } + #[test] + fn should_return_invalid_result_when_entry_payload_is_changed() { + let platform_version = PlatformVersion::latest(); + let data_contract_id = Identifier::random(); + let document_type_name = "note"; + + // Both types are valid indexOnly types over the same properties: + // one keeps `body` in every entry's value slot, the other keys + // by it as the terminal's last component. Every other config + // flag is equal, so `validate_config` reaches the payload check. + let payload_schema = platform_value!({ + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "properties": { + "postId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "position": 0, + }, + "body": { + "type": "array", + "byteArray": true, + "maxItems": 16, + "position": 1, + } + }, + "required": ["postId", "body"], + "indices": [ + { + "name": "byPost", + "properties": [{ "postId": "asc" }], + } + ], + "entryPayload": ["body"], + "additionalProperties": false, + }); + let keyed_schema = platform_value!({ + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "properties": { + "postId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "position": 0, + }, + "body": { + "type": "array", + "byteArray": true, + "maxItems": 16, + "position": 1, + } + }, + "required": ["postId", "body"], + "indices": [ + { + "name": "byPost", + "properties": [{ "postId": "asc" }], + "terminal": ["$ownerId", "body"], + } + ], + "additionalProperties": false, + }); + + let config = DataContractConfig::default_for_version(platform_version) + .expect("should create a default config"); + + let make_document_type = |schema: platform_value::Value| { + DocumentType::try_from_schema( + data_contract_id, + 1, + config.version(), + document_type_name, + schema, + None, + &BTreeMap::new(), + &config, + false, + &mut Vec::new(), + platform_version, + ) + .expect("document type should parse") + }; + + let old_document_type = make_document_type(payload_schema); + let new_document_type = make_document_type(keyed_schema); + + let result = old_document_type + .as_ref() + .validate_config(new_document_type.as_ref()); + + assert_matches!( + result.errors.as_slice(), + [ConsensusError::StateError( + StateError::DocumentTypeUpdateError(e) + )] if e.additional_message().starts_with("document type can not change its entryPayload") + ); + + // The unchanged pair passes the same check. + let result = old_document_type + .as_ref() + .validate_config(old_document_type.as_ref()); + assert!(result.errors.is_empty(), "{:?}", result.errors); + } + #[test] fn should_return_invalid_result_when_range_countable_is_changed() { // documents_countable must remain equal across old/new so that diff --git a/packages/rs-dpp/src/data_contract/document_type/mod.rs b/packages/rs-dpp/src/data_contract/document_type/mod.rs index 5ebdfc67648..158f230179b 100644 --- a/packages/rs-dpp/src/data_contract/document_type/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/mod.rs @@ -143,6 +143,14 @@ pub(crate) mod property_names { /// 14). See `apply_index_only` in `try_from_schema::common` for the /// structural constraints the flag imposes. pub const INDEX_ONLY: &str = "indexOnly"; + /// Doctype-level list, on an `indexOnly` type, of the top-level properties + /// stored in every entry's value (after the row commitment) instead of in + /// a key: the type's value slot. Listed properties must be required, must + /// not appear in any index (as a property or a terminal component) and + /// must be bounded; they are recovered by decoding the proved element. + /// Meta-schema v3+ (protocol version 14). See `apply_index_only` in + /// `try_from_schema::common`. + pub const ENTRY_PAYLOAD: &str = "entryPayload"; /// Doctype-level flag letting the contract's moderators (its owner and the /// identities its moderation config appoints) delete documents of this type /// with a `ContractUserModeration` transition, whatever `canBeDeleted` says diff --git a/packages/rs-dpp/src/data_contract/document_type/property/mod.rs b/packages/rs-dpp/src/data_contract/document_type/property/mod.rs index b50a8562e1a..239f4b9a505 100644 --- a/packages/rs-dpp/src/data_contract/document_type/property/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/property/mod.rs @@ -561,6 +561,39 @@ impl DocumentPropertyType { } } + /// The width every value of this type encodes to as a tree key + /// ([`Self::encode_value_for_tree_keys`]), when that width is fixed: + /// the integer, float, boolean, date and identifier encodings, and a + /// byte array whose bounds pin one size. `None` for strings (their + /// bound counts characters), unbounded or variable-size byte arrays, + /// objects and arrays. The one width the composite indexOnly terminal + /// rules, the walkers and synthesis all split member keys by. + pub fn fixed_tree_key_width(&self) -> Option { + match self { + DocumentPropertyType::U128 | DocumentPropertyType::I128 => Some(16), + DocumentPropertyType::U64 + | DocumentPropertyType::I64 + | DocumentPropertyType::F64 + | DocumentPropertyType::Date => Some(8), + DocumentPropertyType::U32 | DocumentPropertyType::I32 => Some(4), + DocumentPropertyType::U16 | DocumentPropertyType::I16 => Some(2), + DocumentPropertyType::U8 | DocumentPropertyType::I8 | DocumentPropertyType::Boolean => { + Some(1) + } + DocumentPropertyType::Identifier | DocumentPropertyType::IdentifierWithReference(_) => { + Some(32) + } + DocumentPropertyType::ByteArray(sizes) => match (sizes.min_size, sizes.max_size) { + (Some(min), Some(max)) if min == max && min > 0 => Some(min), + _ => None, + }, + DocumentPropertyType::String(_) + | DocumentPropertyType::Object(_) + | DocumentPropertyType::Array(_) + | DocumentPropertyType::VariableTypeArray(_) => None, + } + } + /// The middle size rounded down halfway between min and max size pub fn middle_size(&self, platform_version: &PlatformVersion) -> Option { let min_size = self.min_size()?; diff --git a/packages/rs-dpp/src/data_contract/document_type/v2/accessors.rs b/packages/rs-dpp/src/data_contract/document_type/v2/accessors.rs index 2c9432fdfdb..800e9ee59ea 100644 --- a/packages/rs-dpp/src/data_contract/document_type/v2/accessors.rs +++ b/packages/rs-dpp/src/data_contract/document_type/v2/accessors.rs @@ -232,6 +232,11 @@ impl DocumentTypeV2Getters for DocumentTypeV2 { self.index_only } + /// The entry-payload property names of an indexOnly type. + fn entry_payload(&self) -> &BTreeSet { + &self.entry_payload + } + fn documents_can_be_deleted_by_moderators(&self) -> bool { self.documents_can_be_deleted_by_moderators } diff --git a/packages/rs-dpp/src/data_contract/document_type/v2/mod.rs b/packages/rs-dpp/src/data_contract/document_type/v2/mod.rs index 9bfefc4259e..6c5213eaf44 100644 --- a/packages/rs-dpp/src/data_contract/document_type/v2/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/v2/mod.rs @@ -55,6 +55,10 @@ pub struct DocumentTypeV2 { /// keyword, protocol version 14). Once present they are frozen like the /// rest of the list. Every entry is also in `immutable_fields`. pub(in crate::data_contract) immutable_fields_allow_setting: BTreeSet, + /// On an indexOnly type, the top-level properties stored in every entry's + /// value after the row commitment (the `entryPayload` keyword), in name + /// order. Empty on every other type and on every pre-PV14 contract. + pub(in crate::data_contract) entry_payload: BTreeSet, /// Should documents keep history? pub(in crate::data_contract) documents_keep_history: bool, /// Should transfers of documents of this type be recorded in the document @@ -187,6 +191,7 @@ impl From for DocumentTypeV2 { transient_fields: value.transient_fields, immutable_fields: BTreeSet::new(), immutable_fields_allow_setting: BTreeSet::new(), + entry_payload: BTreeSet::new(), documents_keep_history: value.documents_keep_history, documents_keep_transfer_history: value.documents_keep_transfer_history, documents_keep_purchase_history: value.documents_keep_purchase_history, @@ -232,6 +237,7 @@ impl From for DocumentTypeV2 { transient_fields: value.transient_fields, immutable_fields: BTreeSet::new(), immutable_fields_allow_setting: BTreeSet::new(), + entry_payload: BTreeSet::new(), documents_keep_history: value.documents_keep_history, documents_keep_transfer_history: value.documents_keep_transfer_history, documents_keep_purchase_history: value.documents_keep_purchase_history, diff --git a/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/action_validation/document/document_create_transition_action/state_v1/mod.rs b/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/action_validation/document/document_create_transition_action/state_v1/mod.rs index a7b53e53a74..2943b10fa16 100644 --- a/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/action_validation/document/document_create_transition_action/state_v1/mod.rs +++ b/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/action_validation/document/document_create_transition_action/state_v1/mod.rs @@ -135,7 +135,7 @@ impl DocumentCreateTransitionActionStateValidationV1 for DocumentCreateTransitio .properties .iter() .map(|property| property.name.clone()) - .chain(index.terminal.clone()) + .chain(index.terminal_components().iter().cloned()) .collect(), ), )), diff --git a/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/state/v0/index_only_batch_entries.rs b/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/state/v0/index_only_batch_entries.rs index 74944ddbde5..f34b8266a8e 100644 --- a/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/state/v0/index_only_batch_entries.rs +++ b/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/state/v0/index_only_batch_entries.rs @@ -116,7 +116,7 @@ impl IndexOnlyBatchEntries { .properties .iter() .map(|property| property.name.clone()) - .chain(index.terminal.clone()) + .chain(index.terminal_components().iter().cloned()) .collect(), ) .into(), diff --git a/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/tests/document/index_only.rs b/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/tests/document/index_only.rs index c90252cf451..50d6d71ff29 100644 --- a/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/tests/document/index_only.rs +++ b/packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/tests/document/index_only.rs @@ -2464,4 +2464,370 @@ mod index_only_executed_proof_tests { let (_, absent) = documents.into_iter().next().expect("one entry"); assert!(absent.is_none(), "the deleted tip must be proven absent"); } + + /// The scalar-terminal fixture shared with rs-drive's + /// `index_only_scalar_terminal_e2e_tests`: an `answer` keys its entry + /// by a 33-byte `payload` under `[requestId, $ownerId]`. + const SCALAR_TERMINAL_CONTRACT: &str = "../rs-drive/tests/supporting_files/contract/index-only-scalar-terminal/index-only-scalar-terminal-contract.json"; + + /// A scalar terminal through the full pipeline. The create keys its + /// entry by the 33-byte payload; the executed-create proof locates that + /// entry from the transition's values through the same + /// `serialize_value_for_key` encoding the walker keyed it with, and the + /// verified document carries the payload decoded off the member key; + /// the executed delete proves the entry absent. + #[tokio::test] + async fn test_executed_scalar_terminal_create_and_delete_proofs() { + use dpp::data_contract::accessors::v0::DataContractV0Setters; + let platform_version = PlatformVersion::latest(); + let mut platform = TestPlatformBuilder::new() + .build_with_mock_rpc() + .set_genesis_state(); + let platform_state = platform.state.load(); + let mut rng = StdRng::seed_from_u64(2718); + + let (alice, alice_signer, alice_key) = + setup_identity(&mut platform, 958, dash_to_credits!(1.0)); + let mut contract = + json_document_to_contract(SCALAR_TERMINAL_CONTRACT, true, platform_version) + .expect("expected to parse the scalar-terminal contract"); + contract.set_owner_id(alice.id()); + platform + .drive + .apply_contract( + &contract, + BlockInfo::default(), + true, + StorageFlags::optional_default_as_cow(), + None, + platform_version, + ) + .expect("expected to apply the scalar-terminal contract"); + let answer_type = contract + .document_type_for_name("answer") + .expect("answer doctype exists"); + let contract_arc = Arc::new(contract.clone()); + + let request_id = vec![0x5A; 20]; + let payload = vec![0x7E; 33]; + let entropy = Bytes32::random_with_rng(&mut rng); + let mut answer = answer_type + .random_document_with_identifier_and_entropy( + &mut rng, + alice.id(), + entropy, + DocumentFieldFillType::FillIfNotRequired, + DocumentFieldFillSize::AnyDocumentFillSize, + platform_version, + ) + .expect("expected a random answer"); + answer.set( + "requestId", + dpp::platform_value::Value::Bytes(request_id.clone()), + ); + answer.set( + "payload", + dpp::platform_value::Value::Bytes(payload.clone()), + ); + answer + .set_id_for_creation(answer_type, &entropy.0, 2, platform_version) + .expect("expected to set the document id"); + + let create = BatchTransition::new_document_creation_transition_from_document( + answer.clone(), + answer_type, + entropy.0, + &alice_key, + 2, + 0, + None, + &alice_signer, + platform_version, + None, + ) + .await + .expect("expected the create transition"); + let result = process_and_commit(&platform, &platform_state, &create, platform_version); + assert_eq!( + result.valid_count(), + 1, + "the answer must be created: {:?}", + result.execution_results() + ); + + // ── prove + verify the executed create ───────────────────────── + let proof = platform + .drive + .prove_state_transition(&create, None, platform_version) + .expect("expected to prove the executed create") + .into_data() + .expect("expected proof bytes"); + let lookup = |_id: &dpp::identifier::Identifier| Ok(Some(Arc::clone(&contract_arc))); + let (root_hash, outcome) = Drive::verify_state_transition_was_executed_with_proof( + &create, + &BlockInfo::default(), + proof.as_slice(), + &lookup, + platform_version, + ) + .expect("expected the executed-create proof to verify"); + assert_ne!(root_hash, [0u8; 32]); + assert_matches!( + &outcome, + dpp::state_transition::proof_result::StateTransitionProofOutcome::AffectedState(_) + ); + let StateTransitionProofResult::VerifiedDocuments(documents) = outcome.into_result() else { + panic!("expected verified documents"); + }; + let (_, verified) = documents.into_iter().next().expect("one document"); + let verified = verified.expect("the created answer is present"); + assert_eq!(verified.owner_id(), alice.id()); + assert_eq!( + verified + .properties() + .get("requestId") + .expect("requestId present") + .to_binary_bytes() + .expect("bytes"), + request_id + ); + assert_eq!( + verified + .properties() + .get("payload") + .expect("payload decoded off the member key") + .to_binary_bytes() + .expect("bytes"), + payload + ); + + // ── prove + verify the executed delete ───────────────────────── + let delete = BatchTransition::new_document_deletion_transition_from_document( + answer, + answer_type, + &alice_key, + 3, + 0, + None, + &alice_signer, + platform_version, + None, + ) + .await + .expect("expected the delete transition"); + let result = process_and_commit(&platform, &platform_state, &delete, platform_version); + assert_eq!( + result.valid_count(), + 1, + "the answer must be deleted: {:?}", + result.execution_results() + ); + let proof = platform + .drive + .prove_state_transition(&delete, None, platform_version) + .expect("expected to prove the executed delete") + .into_data() + .expect("expected proof bytes"); + let (root_hash, outcome) = Drive::verify_state_transition_was_executed_with_proof( + &delete, + &BlockInfo::default(), + proof.as_slice(), + &lookup, + platform_version, + ) + .expect("expected the executed-delete proof to verify"); + assert_ne!(root_hash, [0u8; 32]); + assert_matches!( + &outcome, + dpp::state_transition::proof_result::StateTransitionProofOutcome::AffectedState(_) + ); + let StateTransitionProofResult::VerifiedDocuments(documents) = outcome.into_result() else { + panic!("expected verified documents"); + }; + let (_, absent) = documents.into_iter().next().expect("one entry"); + assert!(absent.is_none(), "the deleted answer must be proven absent"); + } + + /// A FLAT composite terminal with an entry payload through the full + /// pipeline: the create's entry sits under the flat level keyed by + /// `hash ‖ owner`, the executed-create proof locates it from the + /// transition's values (the same concatenated encoding the walker + /// keyed it with) and checks the item's leading 32 bytes against the + /// recomputed commitment, and the executed delete proves it absent. + #[tokio::test] + async fn test_executed_flat_composite_create_and_delete_proofs() { + use dpp::data_contract::accessors::v0::DataContractV0Setters; + let platform_version = PlatformVersion::latest(); + let mut platform = TestPlatformBuilder::new() + .build_with_mock_rpc() + .set_genesis_state(); + let platform_state = platform.state.load(); + let mut rng = StdRng::seed_from_u64(3141); + + let (alice, alice_signer, alice_key) = + setup_identity(&mut platform, 958, dash_to_credits!(1.0)); + let mut contract = + json_document_to_contract(SCALAR_TERMINAL_CONTRACT, true, platform_version) + .expect("expected to parse the scalar-terminal contract"); + contract.set_owner_id(alice.id()); + platform + .drive + .apply_contract( + &contract, + BlockInfo::default(), + true, + StorageFlags::optional_default_as_cow(), + None, + platform_version, + ) + .expect("expected to apply the scalar-terminal contract"); + let response_type = contract + .document_type_for_name("loginKeyResponse") + .expect("loginKeyResponse doctype exists"); + let contract_arc = Arc::new(contract.clone()); + + let request_id = vec![0x5A; 20]; + let wallet_key = vec![0x7E; 33]; + let ciphertext = vec![0xC7; 92]; + let entropy = Bytes32::random_with_rng(&mut rng); + let mut answer = response_type + .random_document_with_identifier_and_entropy( + &mut rng, + alice.id(), + entropy, + DocumentFieldFillType::FillIfNotRequired, + DocumentFieldFillSize::AnyDocumentFillSize, + platform_version, + ) + .expect("expected a random response"); + answer.set( + "appEphemeralPubKeyHash", + dpp::platform_value::Value::Bytes(request_id.clone()), + ); + answer.set( + "walletEphemeralPubKey", + dpp::platform_value::Value::Bytes(wallet_key.clone()), + ); + answer.set( + "encryptedPayload", + dpp::platform_value::Value::Bytes(ciphertext.clone()), + ); + answer + .set_id_for_creation(response_type, &entropy.0, 2, platform_version) + .expect("expected to set the document id"); + + let create = BatchTransition::new_document_creation_transition_from_document( + answer.clone(), + response_type, + entropy.0, + &alice_key, + 2, + 0, + None, + &alice_signer, + platform_version, + None, + ) + .await + .expect("expected the create transition"); + let result = process_and_commit(&platform, &platform_state, &create, platform_version); + assert_eq!( + result.valid_count(), + 1, + "the answer must be created: {:?}", + result.execution_results() + ); + + // ── prove + verify the executed create ───────────────────────── + let proof = platform + .drive + .prove_state_transition(&create, None, platform_version) + .expect("expected to prove the executed create") + .into_data() + .expect("expected proof bytes"); + let lookup = |_id: &dpp::identifier::Identifier| Ok(Some(Arc::clone(&contract_arc))); + let (root_hash, outcome) = Drive::verify_state_transition_was_executed_with_proof( + &create, + &BlockInfo::default(), + proof.as_slice(), + &lookup, + platform_version, + ) + .expect("expected the executed-create proof to verify"); + assert_ne!(root_hash, [0u8; 32]); + assert_matches!( + &outcome, + dpp::state_transition::proof_result::StateTransitionProofOutcome::AffectedState(_) + ); + let StateTransitionProofResult::VerifiedDocuments(documents) = outcome.into_result() else { + panic!("expected verified documents"); + }; + let (_, verified) = documents.into_iter().next().expect("one document"); + let verified = verified.expect("the created answer is present"); + assert_eq!(verified.owner_id(), alice.id()); + assert_eq!( + verified + .properties() + .get("appEphemeralPubKeyHash") + .expect("request hash present") + .to_binary_bytes() + .expect("bytes"), + request_id + ); + assert_eq!( + verified + .properties() + .get("encryptedPayload") + .expect("ciphertext present") + .to_binary_bytes() + .expect("bytes"), + ciphertext + ); + + // ── prove + verify the executed delete ───────────────────────── + let delete = BatchTransition::new_document_deletion_transition_from_document( + answer, + response_type, + &alice_key, + 3, + 0, + None, + &alice_signer, + platform_version, + None, + ) + .await + .expect("expected the delete transition"); + let result = process_and_commit(&platform, &platform_state, &delete, platform_version); + assert_eq!( + result.valid_count(), + 1, + "the answer must be deleted: {:?}", + result.execution_results() + ); + let proof = platform + .drive + .prove_state_transition(&delete, None, platform_version) + .expect("expected to prove the executed delete") + .into_data() + .expect("expected proof bytes"); + let (root_hash, outcome) = Drive::verify_state_transition_was_executed_with_proof( + &delete, + &BlockInfo::default(), + proof.as_slice(), + &lookup, + platform_version, + ) + .expect("expected the executed-delete proof to verify"); + assert_ne!(root_hash, [0u8; 32]); + assert_matches!( + &outcome, + dpp::state_transition::proof_result::StateTransitionProofOutcome::AffectedState(_) + ); + let StateTransitionProofResult::VerifiedDocuments(documents) = outcome.into_result() else { + panic!("expected verified documents"); + }; + let (_, absent) = documents.into_iter().next().expect("one entry"); + assert!(absent.is_none(), "the deleted answer must be proven absent"); + } } diff --git a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/index_only_scalar_terminal_e2e_tests.rs b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/index_only_scalar_terminal_e2e_tests.rs new file mode 100644 index 00000000000..8fff41cb91f --- /dev/null +++ b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/index_only_scalar_terminal_e2e_tests.rs @@ -0,0 +1,1783 @@ +//! End-to-end coverage for indexOnly **scalar terminals**: an index's +//! `terminal` may name any property a prefix position admits, not only +//! `$ownerId` or a refersTo identifier. The member key is the terminal +//! value in its tree-key encoding, so a 33-byte byte array, a string and +//! an integer key the `0` member bucket exactly as a prefix level would +//! key by them. +//! +//! Runs against the `index-only-scalar-terminal` fixture at +//! `tests/supporting_files/contract/index-only-scalar-terminal/`: +//! +//! | doctype | index | properties | terminal | terminal type | +//! |----------|-------------|-------------------------|------------|----------------| +//! | `answer` | `byRequest` | `[requestId, $ownerId]` | `payload` | 33-byte array | +//! | `vote` | `byPoll` | `[pollId, $ownerId]` | `choice` | string ≤ 16 | +//! | `vote` | `byChoice` | `[pollId, choice]` | `$ownerId` | ranked count | +//! | `rating` | `byPost` | `[postId, $ownerId]` | `stars` | integer 1 to 5 | +//! | `reaction` | `byPostKind` | `[postId]` | `kind ‖ $ownerId` | composite, prefixed | +//! | `loginKeyResponse` | `byRequest` | (none: flat) | `appEphemeralPubKeyHash ‖ $ownerId` | composite, flat; `entryPayload` | +//! | `note` | `byPostBody` | `[postId]` | `$ownerId ‖ body` | composite; variable-width last | +//! +//! The last two exercise **composite terminals** (the member key is the +//! concatenation of several components' encodings) — one below a prefix +//! level, one FLAT (no prefix at all, entries directly under a level keyed +//! by the terminal's names) — and the flat one also carries an +//! **entry payload**: the wallet key and the ciphertext ride in every +//! entry's item after the row commitment, the type's value slot. +//! +//! Pinned: the entry layout (member key = encoded terminal value, element +//! = row-commitment `Item`), structural uniqueness spanning the terminal +//! value, query synthesis and proof parity with the terminal decoded off +//! the member key, terminal-equality lookups, delete-by-values symmetry, +//! and the fee invariant: the dry run's member-key width follows the +//! terminal property, so the estimate keeps upper-bounding the applied +//! fee for keys narrower and wider than 32 bytes. + +use super::index_only_e2e_tests::{ + assert_grovedb_is_consistent, count_top_k, platform_version, read_grove_element, +}; +use crate::drive::document::query::QueryDocumentsOutcomeV0Methods; +use crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE; +use crate::drive::Drive; +use crate::error::Error; +use crate::query::{DriveDocumentQuery, InternalClauses, OrderClause, WhereClause, WhereOperator}; +use crate::util::object_size_info::DocumentInfo::DocumentRefInfo; +use crate::util::object_size_info::{DocumentAndContractInfo, OwnedDocumentInfo}; +use crate::util::storage_flags::StorageFlags; +use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; +use dpp::block::block_info::BlockInfo; +use dpp::data_contract::accessors::v0::DataContractV0Getters; +use dpp::data_contract::document_type::methods::DocumentTypeV0Methods; +use dpp::data_contract::document_type::random_document::CreateRandomDocument; +use dpp::document::serialization_traits::DocumentPlatformConversionMethodsV0; +use dpp::document::{Document, DocumentV0Getters, DocumentV0Setters}; +use dpp::fee::fee_result::FeeResult; +use dpp::platform_value::{Identifier, Value}; +use dpp::prelude::DataContract; +use dpp::tests::json_document::json_document_to_contract; +use grovedb::Element; +use std::collections::BTreeMap; + +const FIXTURE: &str = "tests/supporting_files/contract/index-only-scalar-terminal/index-only-scalar-terminal-contract.json"; + +const REQUEST: [u8; 20] = [0x5A; 20]; +const PAYLOAD_1: [u8; 33] = [0x01; 33]; +const PAYLOAD_2: [u8; 33] = [0x02; 33]; +const POLL: [u8; 32] = [0xC3; 32]; +const POST: [u8; 32] = [0xD4; 32]; +const OWNER_1: [u8; 32] = [0x11; 32]; +const OWNER_2: [u8; 32] = [0x22; 32]; +const OWNER_3: [u8; 32] = [0x33; 32]; + +/// A fresh Drive with the scalar-terminal fixture contract applied. +fn setup() -> (Drive, DataContract) { + let drive = setup_drive_with_initial_state_structure(None); + let pv = platform_version(); + let contract = + json_document_to_contract(FIXTURE, false, pv).expect("expected to parse the fixture"); + drive + .apply_contract( + &contract, + BlockInfo::default(), + true, + StorageFlags::optional_default_as_cow(), + None, + pv, + ) + .expect("expected to apply the fixture contract"); + (drive, contract) +} + +/// `[DataContractDocuments, contract_id, 1, ]`. +fn doctype_path(contract: &DataContract, doctype: &str) -> Vec> { + vec![ + vec![crate::drive::RootTree::DataContractDocuments as u8], + contract.id().as_bytes().to_vec(), + vec![1], + doctype.as_bytes().to_vec(), + ] +} + +/// A random document of `doctype` with the given properties and owner set. +fn build( + contract: &DataContract, + doctype: &str, + properties: Vec<(&str, Value)>, + owner: [u8; 32], + seed: u64, +) -> Document { + let document_type = contract + .document_type_for_name(doctype) + .expect("doctype exists"); + let mut document = document_type + .random_document(Some(seed), platform_version()) + .expect("random document"); + document.set_properties( + properties + .into_iter() + .map(|(name, value)| (name.to_string(), value)) + .collect::>(), + ); + document.set_owner_id(Identifier::from(owner)); + document +} + +/// An `answer` document: a request hash with its response payload. +fn build_answer( + contract: &DataContract, + request: [u8; 20], + payload: [u8; 33], + owner: [u8; 32], + seed: u64, +) -> Document { + build( + contract, + "answer", + vec![ + ("requestId", Value::Bytes(request.to_vec())), + ("payload", Value::Bytes(payload.to_vec())), + ], + owner, + seed, + ) +} + +/// A `vote` document for `choice` by `owner`. +fn build_vote(contract: &DataContract, choice: &str, owner: [u8; 32], seed: u64) -> Document { + build( + contract, + "vote", + vec![ + ("pollId", Value::Identifier(POLL)), + ("choice", Value::Text(choice.to_string())), + ], + owner, + seed, + ) +} + +/// A `rating` document of `stars` by `owner`. +fn build_rating(contract: &DataContract, stars: u64, owner: [u8; 32], seed: u64) -> Document { + build( + contract, + "rating", + vec![ + ("postId", Value::Identifier(POST)), + ("stars", Value::U64(stars)), + ], + owner, + seed, + ) +} + +/// Inserts (or dry-runs, when `apply` is false) the document and returns its fee. +fn insert( + drive: &Drive, + contract: &DataContract, + doctype: &str, + document: &Document, + apply: bool, +) -> Result { + let document_type = contract + .document_type_for_name(doctype) + .expect("doctype exists"); + drive.add_document_for_contract( + DocumentAndContractInfo { + owned_document_info: OwnedDocumentInfo { + document_info: DocumentRefInfo((document, None)), + owner_id: None, + }, + contract, + document_type, + }, + false, + BlockInfo::default(), + apply, + None, + platform_version(), + None, + ) +} + +/// Deletes (or dry-runs, when `apply` is false) the document and returns its fee. +fn delete( + drive: &Drive, + contract: &DataContract, + doctype: &str, + document: Document, + apply: bool, +) -> Result { + let document_type = contract + .document_type_for_name(doctype) + .expect("doctype exists"); + drive.delete_index_only_document_for_contract( + document, + contract, + document_type, + BlockInfo::default(), + apply, + None, + platform_version(), + None, + ) +} + +/// A query on `doctype` with the given clauses and no ordering. +fn query<'a>( + contract: &'a DataContract, + doctype: &str, + clauses: Vec, + limit: Option, +) -> DriveDocumentQuery<'a> { + let document_type = contract + .document_type_for_name(doctype) + .expect("doctype exists"); + DriveDocumentQuery { + contract, + document_type, + internal_clauses: InternalClauses::extract_from_clauses(clauses, platform_version()) + .expect("clauses extract"), + offset: None, + limit, + order_by: Default::default(), + start_at: None, + start_at_included: false, + block_time_ms: None, + resolved_time_ranges: vec![], + sub_queries: vec![], + } +} + +/// An equality clause on `field`. +fn equal(field: &str, value: Value) -> WhereClause { + WhereClause { + field: field.to_string(), + operator: WhereOperator::Equal, + value, + } +} + +/// The entry under `[…prefix values, 0]` keyed by `member_key`. +fn entry( + drive: &Drive, + contract: &DataContract, + doctype: &str, + prefix: &[(&str, &[u8])], + member_key: &[u8], +) -> Option { + let mut path = doctype_path(contract, doctype); + for (property, value) in prefix { + path.push(property.as_bytes().to_vec()); + path.push(value.to_vec()); + } + path.push(vec![0]); + read_grove_element(drive, &path, member_key) +} + +/// Asserts the element is a bare row-commitment Item (no entry payload). +fn assert_commitment_item(element: Option, what: &str) { + match element { + Some(Element::Item(payload, _)) => assert_eq!( + payload.len(), + INDEX_ONLY_ROW_COMMITMENT_SIZE as usize, + "{what}: the element is the row-commitment Item" + ), + other => panic!("{what}: expected a commitment Item, got {other:?}"), + } +} + +// --------------------------------------------------------------------------- +// Entry layout +// --------------------------------------------------------------------------- + +/// Each doctype registers without a primary-key tree, and an insert keys +/// its member entry by the terminal's tree-key encoding: the raw 33 bytes +/// for the byte array, the UTF-8 bytes for the string, and the property's +/// integer key encoding — none of them 32 bytes wide. +#[test] +fn scalar_terminal_entries_are_keyed_by_the_encoded_terminal_value() { + let (drive, contract) = setup(); + + for doctype in ["answer", "vote", "rating"] { + assert!( + read_grove_element(&drive, &doctype_path(&contract, doctype), &[0]).is_none(), + "{doctype}: an indexOnly doctype has no primary-key tree" + ); + } + + let answer = build_answer(&contract, REQUEST, PAYLOAD_1, OWNER_1, 1); + insert(&drive, &contract, "answer", &answer, true).expect("insert answer"); + assert_commitment_item( + entry( + &drive, + &contract, + "answer", + &[("requestId", &REQUEST), ("$ownerId", &OWNER_1)], + &PAYLOAD_1, + ), + "answer keyed by its 33-byte payload", + ); + + let vote = build_vote(&contract, "yes", OWNER_1, 2); + insert(&drive, &contract, "vote", &vote, true).expect("insert vote"); + assert_commitment_item( + entry( + &drive, + &contract, + "vote", + &[("pollId", &POLL), ("$ownerId", &OWNER_1)], + b"yes", + ), + "vote keyed by its choice's UTF-8 bytes", + ); + // The sibling index keys the same row the ordinary way round: the + // choice in the prefix, the owner as the member key. + assert_commitment_item( + entry( + &drive, + &contract, + "vote", + &[("pollId", &POLL), ("choice", b"yes")], + &OWNER_1, + ), + "vote under byChoice", + ); + + let rating = build_rating(&contract, 4, OWNER_1, 3); + insert(&drive, &contract, "rating", &rating, true).expect("insert rating"); + let rating_type = contract + .document_type_for_name("rating") + .expect("rating doctype exists"); + let stars_key = rating_type + .serialize_value_for_key("stars", &Value::U64(4), platform_version()) + .expect("the query-side encoding of the integer"); + assert_ne!( + stars_key.len(), + 32, + "an integer member key is not identifier-wide" + ); + assert_commitment_item( + entry( + &drive, + &contract, + "rating", + &[("postId", &POST), ("$ownerId", &OWNER_1)], + &stars_key, + ), + "rating keyed by the query-side integer encoding", + ); + + assert_grovedb_is_consistent(&drive); +} + +/// Structural uniqueness spans the terminal value: the same (request, +/// owner, payload) twice is a duplicate, while a second payload by the +/// same owner under the same request is a second entry. +#[test] +fn structural_uniqueness_spans_the_terminal_value() { + let (drive, contract) = setup(); + let first = build_answer(&contract, REQUEST, PAYLOAD_1, OWNER_1, 1); + insert(&drive, &contract, "answer", &first, true).expect("first insert"); + assert!( + insert(&drive, &contract, "answer", &first, true).is_err(), + "the same values twice is a duplicate entry" + ); + let second = build_answer(&contract, REQUEST, PAYLOAD_2, OWNER_1, 2); + insert(&drive, &contract, "answer", &second, true) + .expect("a different terminal value by the same owner is a new entry"); + for payload in [PAYLOAD_1, PAYLOAD_2] { + assert_commitment_item( + entry( + &drive, + &contract, + "answer", + &[("requestId", &REQUEST), ("$ownerId", &OWNER_1)], + &payload, + ), + "both payloads sit under the same prefix", + ); + } + assert_grovedb_is_consistent(&drive); +} + +// --------------------------------------------------------------------------- +// Read surface +// --------------------------------------------------------------------------- + +/// A prefix query synthesizes the terminal off the member key — the +/// 33-byte payload comes back as bytes, the choice as text — with proof +/// parity, and a terminal-equality clause answers "did this owner send +/// this payload" with an existence or absence proof. +#[test] +fn scalar_terminal_queries_synthesize_and_prove() { + let (drive, contract) = setup(); + insert( + &drive, + &contract, + "answer", + &build_answer(&contract, REQUEST, PAYLOAD_1, OWNER_1, 1), + true, + ) + .expect("insert answer 1"); + insert( + &drive, + &contract, + "answer", + &build_answer(&contract, REQUEST, PAYLOAD_2, OWNER_2, 2), + true, + ) + .expect("insert answer 2"); + + // ── every answer to the request ── + let by_request = query( + &contract, + "answer", + vec![equal("requestId", Value::Bytes(REQUEST.to_vec()))], + Some(10), + ); + let outcome = drive + .query_documents(by_request.clone(), None, false, None, None) + .expect("prefix query executes"); + let documents = outcome.documents(); + assert_eq!(documents.len(), 2, "two answers to the request"); + let mut seen: Vec<([u8; 32], Vec)> = documents + .iter() + .map(|document| { + assert_eq!( + document + .properties() + .get("requestId") + .expect("requestId recovered from the path") + .to_binary_bytes() + .expect("bytes"), + REQUEST.to_vec() + ); + let payload = document + .properties() + .get("payload") + .expect("payload recovered from the member key") + .to_binary_bytes() + .expect("bytes"); + (document.owner_id().to_buffer(), payload) + }) + .collect(); + seen.sort(); + assert_eq!( + seen, + vec![(OWNER_1, PAYLOAD_1.to_vec()), (OWNER_2, PAYLOAD_2.to_vec())], + "synthesis must recover every (owner, payload) pair" + ); + let (proof, _) = by_request + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("proof generation"); + let (_root, verified) = by_request + .verify_proof(proof.as_slice(), platform_version()) + .expect("proof verification synthesizes"); + let mut verified_ids: Vec<_> = verified.iter().map(|d| d.id()).collect(); + let mut queried_ids: Vec<_> = documents.iter().map(|d| d.id()).collect(); + verified_ids.sort(); + queried_ids.sort(); + assert_eq!( + verified_ids, queried_ids, + "proved and unproved synthesis agree" + ); + let verified_payloads: Vec> = verified + .iter() + .map(|d| { + d.properties() + .get("payload") + .expect("payload verified") + .to_binary_bytes() + .expect("bytes") + }) + .collect(); + assert!(verified_payloads.contains(&PAYLOAD_1.to_vec())); + assert!(verified_payloads.contains(&PAYLOAD_2.to_vec())); + + // ── terminal equality: did OWNER_1 send PAYLOAD_1 / PAYLOAD_2 ── + let sent = query( + &contract, + "answer", + vec![ + equal("requestId", Value::Bytes(REQUEST.to_vec())), + equal("$ownerId", Value::Identifier(OWNER_1)), + equal("payload", Value::Bytes(PAYLOAD_1.to_vec())), + ], + Some(1), + ); + let outcome = drive + .query_documents(sent.clone(), None, false, None, None) + .expect("terminal-equality query executes"); + assert_eq!(outcome.documents().len(), 1); + assert_eq!(outcome.documents()[0].owner_id().to_buffer(), OWNER_1); + let (proof, _) = sent + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("existence proof generation"); + let (_root, verified) = sent + .verify_proof(proof.as_slice(), platform_version()) + .expect("existence proof verification"); + assert_eq!(verified.len(), 1); + + let not_sent = query( + &contract, + "answer", + vec![ + equal("requestId", Value::Bytes(REQUEST.to_vec())), + equal("$ownerId", Value::Identifier(OWNER_1)), + equal("payload", Value::Bytes(PAYLOAD_2.to_vec())), + ], + Some(1), + ); + let outcome = drive + .query_documents(not_sent.clone(), None, false, None, None) + .expect("negative terminal-equality query executes"); + assert!( + outcome.documents().is_empty(), + "OWNER_1 never sent PAYLOAD_2" + ); + let (proof, _) = not_sent + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("absence proof generation"); + let (_root, verified) = not_sent + .verify_proof(proof.as_slice(), platform_version()) + .expect("absence proof verification"); + assert!(verified.is_empty(), "absence must verify as absence"); + + assert_grovedb_is_consistent(&drive); +} + +/// A string terminal: the poll's votes synthesize with their choices, a +/// terminal-equality clause on the string works, and the sibling ranked +/// index — which keys the same rows by owner — ranks the choices. +#[test] +fn string_terminal_votes_synthesize_and_rank() { + let (drive, contract) = setup(); + for (choice, owner, seed) in [ + ("yes", OWNER_1, 1u64), + ("yes", OWNER_2, 2), + ("no", OWNER_3, 3), + ] { + insert( + &drive, + &contract, + "vote", + &build_vote(&contract, choice, owner, seed), + true, + ) + .expect("insert vote"); + } + + let poll_votes = query( + &contract, + "vote", + vec![equal("pollId", Value::Identifier(POLL))], + Some(10), + ); + let outcome = drive + .query_documents(poll_votes.clone(), None, false, None, None) + .expect("poll query executes"); + let mut seen: Vec<([u8; 32], String)> = outcome + .documents() + .iter() + .map(|document| { + let choice = document + .properties() + .get("choice") + .expect("choice recovered") + .as_text() + .expect("text") + .to_string(); + (document.owner_id().to_buffer(), choice) + }) + .collect(); + seen.sort(); + assert_eq!( + seen, + vec![ + (OWNER_1, "yes".to_string()), + (OWNER_2, "yes".to_string()), + (OWNER_3, "no".to_string()), + ] + ); + let (proof, _) = poll_votes + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("proof generation"); + let (_root, verified) = poll_votes + .verify_proof(proof.as_slice(), platform_version()) + .expect("proof verification"); + assert_eq!(verified.len(), 3); + + let owner_3_voted_no = query( + &contract, + "vote", + vec![ + equal("pollId", Value::Identifier(POLL)), + equal("$ownerId", Value::Identifier(OWNER_3)), + equal("choice", Value::Text("no".to_string())), + ], + Some(1), + ); + let outcome = drive + .query_documents(owner_3_voted_no, None, false, None, None) + .expect("string terminal equality executes"); + assert_eq!(outcome.documents().len(), 1); + + // byChoice ranks the choices by count under the poll. + let mut choice_level = doctype_path(&contract, "vote"); + choice_level.extend([b"pollId".to_vec(), POLL.to_vec(), b"choice".to_vec()]); + assert_eq!( + count_top_k(&drive, &choice_level, 10, true), + vec![(2, b"yes".to_vec()), (1, b"no".to_vec())], + "the sibling ranked index composes with a string-terminal index on the same rows" + ); + + assert_grovedb_is_consistent(&drive); +} + +// --------------------------------------------------------------------------- +// Delete symmetry and fees +// --------------------------------------------------------------------------- + +/// Delete-by-values addresses the entry through its terminal value: a +/// delete carrying a different payload finds no entry and is refused, the +/// right one removes the entry, and a repeat is refused. +#[test] +fn scalar_terminal_delete_removes_the_entry_and_refuses_a_wrong_value() { + let (drive, contract) = setup(); + let stored = build_answer(&contract, REQUEST, PAYLOAD_1, OWNER_1, 1); + insert(&drive, &contract, "answer", &stored, true).expect("insert answer"); + let prefix: [(&str, &[u8]); 2] = [("requestId", &REQUEST), ("$ownerId", &OWNER_1)]; + + let wrong = build_answer(&contract, REQUEST, PAYLOAD_2, OWNER_1, 2); + assert!( + delete(&drive, &contract, "answer", wrong, true).is_err(), + "a delete naming a payload never written finds no entry" + ); + assert_commitment_item( + entry(&drive, &contract, "answer", &prefix, &PAYLOAD_1), + "the stored entry survives the refused delete", + ); + + delete(&drive, &contract, "answer", stored.clone(), true).expect("delete by values"); + assert!( + entry(&drive, &contract, "answer", &prefix, &PAYLOAD_1).is_none(), + "the entry is gone" + ); + assert!( + delete(&drive, &contract, "answer", stored, true).is_err(), + "deleting it again finds nothing" + ); + assert_grovedb_is_consistent(&drive); +} + +/// The dry run sizes the member key by the terminal property, so the +/// estimate upper-bounds the applied fee for a 33-byte key, a string key +/// and an integer key alike, on insert and on delete. +#[test] +fn scalar_terminal_estimated_fees_upper_bound_actual_fees() { + let (drive, contract) = setup(); + let cases: Vec<(&str, Document)> = vec![ + ( + "answer", + build_answer(&contract, REQUEST, PAYLOAD_1, OWNER_1, 1), + ), + ("vote", build_vote(&contract, "yes", OWNER_1, 2)), + ("rating", build_rating(&contract, 4, OWNER_1, 3)), + ]; + for (doctype, document) in cases { + let estimated_insert = + insert(&drive, &contract, doctype, &document, false).expect("estimated insert"); + let actual_insert = + insert(&drive, &contract, doctype, &document, true).expect("actual insert"); + assert!( + estimated_insert.storage_fee >= actual_insert.storage_fee, + "{doctype}: estimated insert storage fee {} must upper-bound actual {}", + estimated_insert.storage_fee, + actual_insert.storage_fee + ); + + let estimated_delete = + delete(&drive, &contract, doctype, document.clone(), false).expect("estimated delete"); + let actual_delete = + delete(&drive, &contract, doctype, document, true).expect("actual delete"); + assert!(actual_delete.processing_fee > 0); + assert!( + estimated_delete.processing_fee >= actual_delete.processing_fee, + "{doctype}: estimated delete processing fee {} must upper-bound actual {}", + estimated_delete.processing_fee, + actual_delete.processing_fee + ); + } + assert_grovedb_is_consistent(&drive); +} + +// --------------------------------------------------------------------------- +// Composite terminals and the entry payload +// --------------------------------------------------------------------------- + +const REQUEST_HASH: [u8; 20] = [0x7A; 20]; +const OTHER_REQUEST_HASH: [u8; 20] = [0x7B; 20]; +const WALLET_KEY_1: [u8; 33] = [0xA1; 33]; +const WALLET_KEY_2: [u8; 33] = [0xA2; 33]; +const WALLET_KEY_3: [u8; 33] = [0xA3; 33]; +/// The flat level of `loginKeyResponse.byRequest`: a zero byte, then each +/// terminal component name preceded by a zero byte. +const LOGIN_FLAT_LEVEL: &[u8] = b"\0appEphemeralPubKeyHash\0$ownerId"; + +/// A stand-in ciphertext of `len` copies of `byte`. +fn cipher(byte: u8, len: usize) -> Vec { + vec![byte; len] +} + +/// A `loginKeyResponse` document for `request` with the wallet key and ciphertext payload. +fn build_login_response( + contract: &DataContract, + request: [u8; 20], + wallet_key: [u8; 33], + ciphertext: Vec, + owner: [u8; 32], + seed: u64, +) -> Document { + build( + contract, + "loginKeyResponse", + vec![ + ("appEphemeralPubKeyHash", Value::Bytes(request.to_vec())), + ("walletEphemeralPubKey", Value::Bytes(wallet_key.to_vec())), + ("encryptedPayload", Value::Bytes(ciphertext)), + ], + owner, + seed, + ) +} + +/// A `reaction` document of `kind` on the fixture post by `owner`. +fn build_reaction(contract: &DataContract, kind: u64, owner: [u8; 32], seed: u64) -> Document { + build( + contract, + "reaction", + vec![ + ("postId", Value::Identifier(POST)), + ("kind", Value::U64(kind)), + ], + owner, + seed, + ) +} + +/// A query on `doctype` with the given clauses and `order_by` (field, ascending). +fn query_ordered<'a>( + contract: &'a DataContract, + doctype: &str, + clauses: Vec, + order_by: Vec<(&str, bool)>, + limit: Option, +) -> DriveDocumentQuery<'a> { + let mut query = query(contract, doctype, clauses, limit); + query.order_by = order_by + .into_iter() + .map(|(field, ascending)| { + ( + field.to_string(), + OrderClause { + field: field.to_string(), + ascending, + }, + ) + }) + .collect(); + query +} + +/// The flat member key of a login response: request hash then owner id. +fn login_member_key(request: [u8; 20], owner: [u8; 32]) -> Vec { + let mut key = request.to_vec(); + key.extend(owner); + key +} + +/// Reads the login response entry stored under the flat level for `request` and `owner`. +fn login_entry( + drive: &Drive, + contract: &DataContract, + request: [u8; 20], + owner: [u8; 32], +) -> Option { + let mut path = doctype_path(contract, "loginKeyResponse"); + path.push(LOGIN_FLAT_LEVEL.to_vec()); + path.push(vec![0]); + read_grove_element(drive, &path, &login_member_key(request, owner)) +} + +/// The binary bytes of the document property `name`. +fn payload_bytes(document: &Document, name: &str) -> Vec { + document + .properties() + .get(name) + .unwrap_or_else(|| panic!("{name} present")) + .to_binary_bytes() + .expect("bytes") +} + +/// A flat composite index writes its entry directly under its own level: +/// `[…, "\0appEphemeralPubKeyHash\0$ownerId", 0, hash ‖ owner]`, no +/// property-name tree and no value tree for either component. The item is +/// the row commitment followed by the entry payload — each payload property +/// in name order, length-framed: the ciphertext first, then the wallet key. +#[test] +fn flat_composite_entries_carry_the_payload_in_the_item() { + let (drive, contract) = setup(); + let ciphertext = cipher(0xC1, 60); + let response = build_login_response( + &contract, + REQUEST_HASH, + WALLET_KEY_1, + ciphertext.clone(), + OWNER_1, + 1, + ); + insert(&drive, &contract, "loginKeyResponse", &response, true).expect("insert response"); + + match login_entry(&drive, &contract, REQUEST_HASH, OWNER_1) { + Some(Element::Item(bytes, _)) => { + let commitment_size = INDEX_ONLY_ROW_COMMITMENT_SIZE as usize; + assert_eq!(bytes.len(), commitment_size + 2 + 60 + 2 + 33); + let mut cursor = commitment_size; + assert_eq!(&bytes[cursor..cursor + 2], &60u16.to_be_bytes()); + cursor += 2; + assert_eq!(&bytes[cursor..cursor + 60], ciphertext.as_slice()); + cursor += 60; + assert_eq!(&bytes[cursor..cursor + 2], &33u16.to_be_bytes()); + cursor += 2; + assert_eq!(&bytes[cursor..], &WALLET_KEY_1); + } + other => panic!("expected the payload-bearing Item, got {other:?}"), + } + let base = doctype_path(&contract, "loginKeyResponse"); + assert!( + read_grove_element(&drive, &base, b"appEphemeralPubKeyHash").is_none() + && read_grove_element(&drive, &base, b"$ownerId").is_none(), + "a flat index owns no property-name trees" + ); + assert!( + read_grove_element(&drive, &base, LOGIN_FLAT_LEVEL).is_some(), + "the flat level tree sits directly under the doctype" + ); + + // The same values by the same owner are a duplicate; another owner is + // a second entry under the same request hash. + assert!(insert(&drive, &contract, "loginKeyResponse", &response, true).is_err()); + insert( + &drive, + &contract, + "loginKeyResponse", + &build_login_response( + &contract, + REQUEST_HASH, + WALLET_KEY_2, + cipher(0xC2, 92), + OWNER_2, + 2, + ), + true, + ) + .expect("a second responder"); + assert!(login_entry(&drive, &contract, REQUEST_HASH, OWNER_2).is_some()); + + assert_grovedb_is_consistent(&drive); +} + +/// The app's read: equality on the request hash (the leading component) +/// returns every responder's owner id with the wallet key and ciphertext +/// decoded off the item, as one proof; a point lookup on hash and owner +/// answers with existence or absence; a clause-free query scans the flat +/// level. +#[test] +fn flat_composite_lookup_by_request_hash_synthesizes_the_payload_and_proves() { + let (drive, contract) = setup(); + let cipher_1 = cipher(0xC1, 60); + let cipher_2 = cipher(0xC2, 572); + for (request, key, ciphertext, owner, seed) in [ + (REQUEST_HASH, WALLET_KEY_1, cipher_1.clone(), OWNER_1, 1u64), + (REQUEST_HASH, WALLET_KEY_2, cipher_2.clone(), OWNER_2, 2), + ( + OTHER_REQUEST_HASH, + WALLET_KEY_3, + cipher(0xC3, 100), + OWNER_3, + 3, + ), + ] { + insert( + &drive, + &contract, + "loginKeyResponse", + &build_login_response(&contract, request, key, ciphertext, owner, seed), + true, + ) + .expect("insert response"); + } + + // ── every answer to the request ── + let by_request = query( + &contract, + "loginKeyResponse", + vec![equal( + "appEphemeralPubKeyHash", + Value::Bytes(REQUEST_HASH.to_vec()), + )], + Some(10), + ); + let outcome = drive + .query_documents(by_request.clone(), None, false, None, None) + .expect("lookup by request hash executes"); + let documents = outcome.documents(); + assert_eq!(documents.len(), 2, "two responders to the request"); + let mut seen: Vec<([u8; 32], Vec, Vec)> = documents + .iter() + .map(|document| { + assert_eq!( + payload_bytes(document, "appEphemeralPubKeyHash"), + REQUEST_HASH.to_vec() + ); + ( + document.owner_id().to_buffer(), + payload_bytes(document, "walletEphemeralPubKey"), + payload_bytes(document, "encryptedPayload"), + ) + }) + .collect(); + seen.sort(); + assert_eq!( + seen, + vec![ + (OWNER_1, WALLET_KEY_1.to_vec(), cipher_1.clone()), + (OWNER_2, WALLET_KEY_2.to_vec(), cipher_2.clone()), + ], + "the owner comes off the member key, the wallet key and ciphertext off the item" + ); + let (proof, _) = by_request + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("proof generation"); + let (_root, verified) = by_request + .verify_proof(proof.as_slice(), platform_version()) + .expect("proof verification synthesizes"); + let mut verified_ids: Vec<_> = verified.iter().map(|d| d.id()).collect(); + let mut queried_ids: Vec<_> = documents.iter().map(|d| d.id()).collect(); + verified_ids.sort(); + queried_ids.sort(); + assert_eq!( + verified_ids, queried_ids, + "proved and unproved synthesis agree" + ); + let verified_ciphers: Vec> = verified + .iter() + .map(|d| payload_bytes(d, "encryptedPayload")) + .collect(); + assert!(verified_ciphers.contains(&cipher_1) && verified_ciphers.contains(&cipher_2)); + + // ── point lookup: did OWNER_1 / OWNER_3 answer this request ── + let answered = query( + &contract, + "loginKeyResponse", + vec![ + equal( + "appEphemeralPubKeyHash", + Value::Bytes(REQUEST_HASH.to_vec()), + ), + equal("$ownerId", Value::Identifier(OWNER_1)), + ], + Some(1), + ); + let outcome = drive + .query_documents(answered.clone(), None, false, None, None) + .expect("point lookup executes"); + assert_eq!(outcome.documents().len(), 1); + assert_eq!( + payload_bytes(&outcome.documents()[0], "walletEphemeralPubKey"), + WALLET_KEY_1.to_vec() + ); + let (proof, _) = answered + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("existence proof generation"); + let (_root, verified) = answered + .verify_proof(proof.as_slice(), platform_version()) + .expect("existence proof verification"); + assert_eq!(verified.len(), 1); + + let not_answered = query( + &contract, + "loginKeyResponse", + vec![ + equal( + "appEphemeralPubKeyHash", + Value::Bytes(REQUEST_HASH.to_vec()), + ), + equal("$ownerId", Value::Identifier(OWNER_3)), + ], + Some(1), + ); + assert!(drive + .query_documents(not_answered.clone(), None, false, None, None) + .expect("negative point lookup executes") + .documents() + .is_empty()); + let (proof, _) = not_answered + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("absence proof generation"); + let (_root, verified) = not_answered + .verify_proof(proof.as_slice(), platform_version()) + .expect("absence proof verification"); + assert!(verified.is_empty(), "absence must verify as absence"); + + // ── everything: a clause-free query scans the flat level ── + let everything = query(&contract, "loginKeyResponse", vec![], Some(10)); + let outcome = drive + .query_documents(everything.clone(), None, false, None, None) + .expect("flat scan executes"); + assert_eq!(outcome.documents().len(), 3); + let (proof, _) = everything + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("flat scan proof generation"); + let (_root, verified) = everything + .verify_proof(proof.as_slice(), platform_version()) + .expect("flat scan proof verification"); + assert_eq!(verified.len(), 3); + + assert_grovedb_is_consistent(&drive); +} + +/// Keyset pagination over the second component: with the request hash +/// bound, `$ownerId > ` ordered by `$ownerId` walks the +/// responders page by page, each page agreeing with its proof. +#[test] +fn flat_composite_keyset_pagination_over_the_second_component() { + let (drive, contract) = setup(); + for (key, owner, seed) in [ + (WALLET_KEY_1, OWNER_1, 1u64), + (WALLET_KEY_2, OWNER_2, 2), + (WALLET_KEY_3, OWNER_3, 3), + ] { + insert( + &drive, + &contract, + "loginKeyResponse", + &build_login_response( + &contract, + REQUEST_HASH, + key, + cipher(seed as u8, 60), + owner, + seed, + ), + true, + ) + .expect("insert response"); + } + + let page_1 = query_ordered( + &contract, + "loginKeyResponse", + vec![equal( + "appEphemeralPubKeyHash", + Value::Bytes(REQUEST_HASH.to_vec()), + )], + vec![("$ownerId", true)], + Some(2), + ); + let outcome = drive + .query_documents(page_1.clone(), None, false, None, None) + .expect("page 1 executes"); + let owners: Vec<[u8; 32]> = outcome + .documents() + .iter() + .map(|d| d.owner_id().to_buffer()) + .collect(); + assert_eq!(owners, vec![OWNER_1, OWNER_2]); + let (proof, _) = page_1 + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("page 1 proof"); + let (_root, verified) = page_1 + .verify_proof(proof.as_slice(), platform_version()) + .expect("page 1 verifies"); + assert_eq!(verified.len(), 2); + + let page_2 = query_ordered( + &contract, + "loginKeyResponse", + vec![ + equal( + "appEphemeralPubKeyHash", + Value::Bytes(REQUEST_HASH.to_vec()), + ), + WhereClause { + field: "$ownerId".to_string(), + operator: WhereOperator::GreaterThan, + value: Value::Identifier(OWNER_2), + }, + ], + vec![("$ownerId", true)], + Some(2), + ); + let outcome = drive + .query_documents(page_2.clone(), None, false, None, None) + .expect("page 2 executes"); + let owners: Vec<[u8; 32]> = outcome + .documents() + .iter() + .map(|d| d.owner_id().to_buffer()) + .collect(); + assert_eq!(owners, vec![OWNER_3]); + assert_eq!( + payload_bytes(&outcome.documents()[0], "walletEphemeralPubKey"), + WALLET_KEY_3.to_vec() + ); + let (proof, _) = page_2 + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("page 2 proof"); + let (_root, verified) = page_2 + .verify_proof(proof.as_slice(), platform_version()) + .expect("page 2 verifies"); + assert_eq!(verified.len(), 1); + + assert_grovedb_is_consistent(&drive); +} + +/// Delete-by-values on a flat composite entry with a payload: a delete +/// carrying a different ciphertext recomputes a different commitment and +/// is refused, the right one removes the entry, and the dry run +/// upper-bounds the applied fee on insert and delete (the item is sized by +/// the payload bound, the key by the components' widths). +#[test] +fn flat_composite_delete_and_fees() { + let (drive, contract) = setup(); + let stored = build_login_response( + &contract, + REQUEST_HASH, + WALLET_KEY_1, + cipher(0xC1, 572), + OWNER_1, + 1, + ); + let estimated_insert = + insert(&drive, &contract, "loginKeyResponse", &stored, false).expect("estimated insert"); + let actual_insert = + insert(&drive, &contract, "loginKeyResponse", &stored, true).expect("actual insert"); + assert!( + estimated_insert.storage_fee >= actual_insert.storage_fee, + "estimated insert storage fee {} must upper-bound actual {}", + estimated_insert.storage_fee, + actual_insert.storage_fee + ); + + let wrong = build_login_response( + &contract, + REQUEST_HASH, + WALLET_KEY_1, + cipher(0xC9, 572), + OWNER_1, + 2, + ); + assert!( + delete(&drive, &contract, "loginKeyResponse", wrong, true).is_err(), + "a delete whose payload disagrees with the stored entry fails the commitment probe" + ); + assert!(login_entry(&drive, &contract, REQUEST_HASH, OWNER_1).is_some()); + + let estimated_delete = delete(&drive, &contract, "loginKeyResponse", stored.clone(), false) + .expect("estimated delete"); + let actual_delete = + delete(&drive, &contract, "loginKeyResponse", stored, true).expect("actual delete"); + assert!(actual_delete.processing_fee > 0); + assert!( + estimated_delete.processing_fee >= actual_delete.processing_fee, + "estimated delete processing fee {} must upper-bound actual {}", + estimated_delete.processing_fee, + actual_delete.processing_fee + ); + assert!(login_entry(&drive, &contract, REQUEST_HASH, OWNER_1).is_none()); + // The flat level survives the last entry's removal: it is registration + // structure, and the next insert lands under it again. + assert!(read_grove_element( + &drive, + &doctype_path(&contract, "loginKeyResponse"), + LOGIN_FLAT_LEVEL + ) + .is_some()); + insert( + &drive, + &contract, + "loginKeyResponse", + &build_login_response( + &contract, + REQUEST_HASH, + WALLET_KEY_2, + cipher(0xC2, 60), + OWNER_2, + 3, + ), + true, + ) + .expect("insert after the level was drained"); + + assert_grovedb_is_consistent(&drive); +} + +/// A composite terminal BELOW a prefix level: `[postId] → kind ‖ $ownerId`. +/// The entry sits in the post's `0` bucket keyed by the integer's tree-key +/// encoding followed by the owner; equality on `kind` (the first +/// component) with the post bound lowers onto a key range and returns the +/// reacting owners; the hierarchical machinery above is untouched. +#[test] +fn prefixed_composite_terminal_ranges_over_the_leading_component() { + let (drive, contract) = setup(); + for (kind, owner, seed) in [(3u64, OWNER_1, 1u64), (3, OWNER_2, 2), (7, OWNER_3, 3)] { + insert( + &drive, + &contract, + "reaction", + &build_reaction(&contract, kind, owner, seed), + true, + ) + .expect("insert reaction"); + } + let reaction_type = contract + .document_type_for_name("reaction") + .expect("reaction doctype exists"); + let mut expected_key = reaction_type + .serialize_value_for_key("kind", &Value::U64(3), platform_version()) + .expect("kind encodes"); + expected_key.extend(OWNER_1); + assert_commitment_item( + entry( + &drive, + &contract, + "reaction", + &[("postId", &POST)], + &expected_key, + ), + "reaction keyed by kind ‖ owner under the post", + ); + + let thumbs = query( + &contract, + "reaction", + vec![ + equal("postId", Value::Identifier(POST)), + equal("kind", Value::U64(3)), + ], + Some(10), + ); + let outcome = drive + .query_documents(thumbs.clone(), None, false, None, None) + .expect("leading-component equality executes"); + let mut owners: Vec<[u8; 32]> = outcome + .documents() + .iter() + .map(|d| d.owner_id().to_buffer()) + .collect(); + owners.sort(); + assert_eq!(owners, vec![OWNER_1, OWNER_2]); + for document in outcome.documents() { + assert_eq!( + document + .properties() + .get("kind") + .and_then(|v| v.to_integer::().ok()), + Some(3), + "the leading component is decoded off the member key" + ); + } + let (proof, _) = thumbs + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("proof generation"); + let (_root, verified) = thumbs + .verify_proof(proof.as_slice(), platform_version()) + .expect("proof verification"); + assert_eq!(verified.len(), 2); + + let all_reactions = query( + &contract, + "reaction", + vec![equal("postId", Value::Identifier(POST))], + Some(10), + ); + let outcome = drive + .query_documents(all_reactions, None, false, None, None) + .expect("prefix query executes"); + assert_eq!(outcome.documents().len(), 3); + + assert_grovedb_is_consistent(&drive); +} + +#[test] +fn should_reject_unrepresentable_composite_terminal_ordering() { + let (drive, contract) = setup(); + for (kind, owner, seed) in [(3, OWNER_1, 1), (3, OWNER_3, 2), (7, OWNER_2, 3)] { + insert( + &drive, + &contract, + "reaction", + &build_reaction(&contract, kind, owner, seed), + true, + ) + .expect("insert reaction"); + } + let (valid_proof, _) = query_ordered( + &contract, + "reaction", + vec![equal("postId", Value::Identifier(POST))], + vec![("kind", true), ("$ownerId", true)], + Some(10), + ) + .execute_with_proof(&drive, None, None, platform_version()) + .expect("prove the actual member-key order"); + for order_by in [ + vec![("$ownerId", true)], + vec![("$ownerId", true), ("kind", true)], + vec![("kind", true), ("$ownerId", false)], + ] { + let query = query_ordered( + &contract, + "reaction", + vec![equal("postId", Value::Identifier(POST))], + order_by, + Some(10), + ); + // A run of components out of declared order is refused by the + // matcher itself (no index can serve it: "valid indexes are"); + // the other shapes reach the terminal route and fail its orderBy + // rule. Either way the executor, the prover and the verifier + // refuse identically. + let unrepresentable = |error: &Error| { + error.to_string().contains("orderBy") || error.to_string().contains("valid indexes are") + }; + let error = drive + .query_documents(query.clone(), None, false, None, None) + .expect_err("one member-key walk cannot implement this ordering"); + assert!(unrepresentable(&error), "{error}"); + let error = query + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect_err("the prover must reject the same ordering"); + assert!(unrepresentable(&error), "{error}"); + let error = query + .verify_proof(&valid_proof, platform_version()) + .expect_err("a proof of member-key order cannot prove a different sort order"); + assert!(unrepresentable(&error), "{error}"); + } +} + +#[test] +fn should_order_composite_terminal_components_in_both_directions() { + let (drive, contract) = setup(); + for (kind, owner, seed) in [(3, OWNER_1, 1), (3, OWNER_3, 2), (7, OWNER_2, 3)] { + insert( + &drive, + &contract, + "reaction", + &build_reaction(&contract, kind, owner, seed), + true, + ) + .expect("insert reaction"); + } + for (order_by, kind, expected) in [ + ( + vec![("kind", true), ("$ownerId", true)], + None, + vec![OWNER_1, OWNER_3, OWNER_2], + ), + ( + vec![("kind", false), ("$ownerId", false)], + None, + vec![OWNER_2, OWNER_3, OWNER_1], + ), + // Equality-bound components do not affect ordering, so their + // direction need not agree with the remaining member-key order. + ( + vec![("kind", true), ("$ownerId", false)], + Some(3), + vec![OWNER_3, OWNER_1], + ), + (vec![("$ownerId", true)], Some(3), vec![OWNER_1, OWNER_3]), + ] { + let mut clauses = vec![equal("postId", Value::Identifier(POST))]; + if let Some(kind) = kind { + clauses.push(equal("kind", Value::U64(kind))); + } + let query = query_ordered(&contract, "reaction", clauses, order_by, Some(10)); + let outcome = drive + .query_documents(query.clone(), None, false, None, None) + .expect("representable ordering executes"); + let owners: Vec<_> = outcome + .documents() + .iter() + .map(|d| d.owner_id().to_buffer()) + .collect(); + assert_eq!(owners, expected); + let (proof, _) = query + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("prove representable ordering"); + let (_, verified) = query + .verify_proof(&proof, platform_version()) + .expect("verify ordering"); + let owners: Vec<_> = verified.iter().map(|d| d.owner_id().to_buffer()).collect(); + assert_eq!(owners, expected); + } +} + +#[test] +fn should_roundtrip_empty_and_nul_entry_payloads_and_bind_deletes() { + let (drive, contract) = setup(); + // Full contract validation admits both zero-length payloads and a NUL + // string. Neither needs the sentinels used for property tree keys. + json_document_to_contract(FIXTURE, true, platform_version()).expect("fixture validates"); + for (text, bytes) in [("", vec![]), ("\0", vec![]), ("héllo", vec![0, 1])] { + let document = build( + &contract, + "payloadValues", + vec![ + ("bytes", Value::Bytes(bytes)), + ("text", Value::Text(text.to_string())), + ], + OWNER_1, + 1, + ); + insert(&drive, &contract, "payloadValues", &document, true).expect("insert payload"); + let query = query(&contract, "payloadValues", vec![], Some(10)); + let (serialized, _, _) = query + .execute_raw_results_no_proof(&drive, None, None, platform_version()) + .expect("a covering flat scan serializes its payload"); + assert_eq!(serialized.len(), 1); + let decoded = Document::from_bytes(&serialized[0], query.document_type, platform_version()) + .expect("deserialize response"); + assert_eq!(decoded.properties(), document.properties()); + let (proof, _) = query + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("prove payload"); + let (_, verified) = query + .verify_proof(&proof, platform_version()) + .expect("verify payload"); + assert_eq!(verified.len(), 1); + assert_eq!(verified[0].properties(), document.properties()); + + let mut wrong = document.clone(); + let mut properties = wrong.properties().clone(); + properties.insert( + "text".to_string(), + Value::Text(if text.is_empty() { "\0" } else { "" }.to_string()), + ); + wrong.set_properties(properties); + assert!( + delete(&drive, &contract, "payloadValues", wrong, true).is_err(), + "empty and NUL strings must have different commitments" + ); + delete(&drive, &contract, "payloadValues", document, true).expect("delete exact payload"); + } + assert_grovedb_is_consistent(&drive); +} + +#[test] +fn should_refuse_serializing_an_incomplete_flat_scan() { + let (drive, contract) = setup(); + let document = build( + &contract, + "tagged", + vec![("tag", Value::Text("hello".to_string()))], + OWNER_1, + 1, + ); + insert(&drive, &contract, "tagged", &document, true).expect("insert tagged document"); + let query = query(&contract, "tagged", vec![], Some(10)); + let error = query + .execute_raw_results_no_proof(&drive, None, None, platform_version()) + .expect_err("the flat projection cannot assert that the stored tag is absent"); + assert!( + error.to_string().contains("does not cover every property"), + "{error}" + ); + + // A proof still exposes an explicit projection, while querying the + // covering tag index may return a complete serialized document. + let (proof, _) = query + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("prove partial flat projection"); + let (_, projected) = query + .verify_proof(&proof, platform_version()) + .expect("verify projection"); + assert_eq!(projected.len(), 1); + assert_eq!(projected[0].owner_id(), document.owner_id()); + assert!(!projected[0].properties().contains_key("tag")); + + let covering = self::query( + &contract, + "tagged", + vec![equal("tag", Value::Text("hello".to_string()))], + Some(10), + ); + let (serialized, _, _) = covering + .execute_raw_results_no_proof(&drive, None, None, platform_version()) + .expect("the covering index can serialize the tag"); + let decoded = Document::from_bytes(&serialized[0], covering.document_type, platform_version()) + .expect("deserialize covering response"); + assert_eq!(decoded.properties(), document.properties()); +} + +/// A `note` document of `body` on the fixture post by `owner`. +fn build_note(contract: &DataContract, body: &[u8], owner: [u8; 32], seed: u64) -> Document { + build( + contract, + "note", + vec![ + ("postId", Value::Identifier(POST)), + ("body", Value::Bytes(body.to_vec())), + ], + owner, + seed, + ) +} + +/// A variable-width LAST component (a byte array with no `minItems`): an +/// empty value contributes no key bytes, so the entry is keyed by the +/// leading component alone and must still synthesize as an empty byte +/// array rather than the tree-key null sentinel. Ranges on the last +/// component are lowered against the key itself (no padding), in both +/// directions of the bound, with proof parity. +#[test] +fn variable_width_last_component_ranges_and_empty_values() { + let (drive, contract) = setup(); + let bodies: [&[u8]; 4] = [b"", b"apple", b"banana", b"cherry"]; + for (seed, body) in bodies.iter().enumerate() { + insert( + &drive, + &contract, + "note", + &build_note(&contract, body, OWNER_1, seed as u64 + 1), + true, + ) + .expect("insert note"); + } + insert( + &drive, + &contract, + "note", + &build_note(&contract, b"zebra", OWNER_2, 9), + true, + ) + .expect("insert another owner's note"); + assert_commitment_item( + entry(&drive, &contract, "note", &[("postId", &POST)], &OWNER_1), + "an empty body keys the entry by the owner alone", + ); + + let run = |clauses: Vec, what: &str| -> Vec> { + let mut clauses = clauses; + clauses.insert(0, equal("postId", Value::Identifier(POST))); + clauses.insert(1, equal("$ownerId", Value::Identifier(OWNER_1))); + let query = query_ordered(&contract, "note", clauses, vec![("body", true)], Some(10)); + let outcome = drive + .query_documents(query.clone(), None, false, None, None) + .unwrap_or_else(|e| panic!("{what}: query executes: {e}")); + let bodies: Vec> = outcome + .documents() + .iter() + .map(|document| payload_bytes(document, "body")) + .collect(); + let (proof, _) = query + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .unwrap_or_else(|e| panic!("{what}: proof generation: {e}")); + let (_root, verified) = query + .verify_proof(proof.as_slice(), platform_version()) + .unwrap_or_else(|e| panic!("{what}: proof verification: {e}")); + let verified_bodies: Vec> = verified + .iter() + .map(|document| payload_bytes(document, "body")) + .collect(); + assert_eq!( + verified_bodies, bodies, + "{what}: proved and unproved synthesis agree" + ); + bodies + }; + let between = |low: &[u8], high: &[u8], operator: WhereOperator| WhereClause { + field: "body".to_string(), + operator, + value: Value::Array(vec![ + Value::Bytes(low.to_vec()), + Value::Bytes(high.to_vec()), + ]), + }; + let bound = |operator: WhereOperator, value: &[u8]| WhereClause { + field: "body".to_string(), + operator, + value: Value::Bytes(value.to_vec()), + }; + + assert_eq!( + run(vec![], "every body of the owner"), + bodies.iter().map(|body| body.to_vec()).collect::>(), + "the empty body is the owner's first key and comes back as an empty array" + ); + assert_eq!( + run(vec![bound(WhereOperator::LessThan, b"b")], "bodies below b"), + vec![Vec::new(), b"apple".to_vec()] + ); + assert_eq!( + run( + vec![between(b"b", b"cz", WhereOperator::Between)], + "bodies between b and cz" + ), + vec![b"banana".to_vec(), b"cherry".to_vec()] + ); + assert_eq!( + run( + vec![between( + b"apple", + b"cherry", + WhereOperator::BetweenExcludeBounds + )], + "bodies strictly between apple and cherry" + ), + vec![b"banana".to_vec()] + ); + assert_eq!( + run( + vec![bound(WhereOperator::GreaterThanOrEquals, b"cherry")], + "bodies from cherry" + ), + vec![b"cherry".to_vec()], + "another owner's later body sits under a different leading component" + ); + + assert_grovedb_is_consistent(&drive); +} + +/// An `in` clause on a composite tail: on the LAST component it addresses +/// each key directly, on a leading component it covers every key under +/// each value; both synthesize the components off the member key with +/// proof parity. +#[test] +fn composite_tail_in_clauses_on_last_and_leading_components() { + let (drive, contract) = setup(); + for (seed, body) in [b"apple".as_slice(), b"banana", b"cherry"] + .iter() + .enumerate() + { + insert( + &drive, + &contract, + "note", + &build_note(&contract, body, OWNER_1, seed as u64 + 1), + true, + ) + .expect("insert note"); + } + for (kind, owner, seed) in [(3u64, OWNER_1, 11u64), (5, OWNER_2, 12), (7, OWNER_3, 13)] { + insert( + &drive, + &contract, + "reaction", + &build_reaction(&contract, kind, owner, seed), + true, + ) + .expect("insert reaction"); + } + + let notes = query_ordered( + &contract, + "note", + vec![ + equal("postId", Value::Identifier(POST)), + equal("$ownerId", Value::Identifier(OWNER_1)), + WhereClause { + field: "body".to_string(), + operator: WhereOperator::In, + value: Value::Array(vec![ + Value::Bytes(b"cherry".to_vec()), + Value::Bytes(b"apple".to_vec()), + ]), + }, + ], + vec![("body", true)], + Some(10), + ); + let outcome = drive + .query_documents(notes.clone(), None, false, None, None) + .expect("in on the last component executes"); + let bodies: Vec> = outcome + .documents() + .iter() + .map(|document| payload_bytes(document, "body")) + .collect(); + assert_eq!(bodies, vec![b"apple".to_vec(), b"cherry".to_vec()]); + let (proof, _) = notes + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("proof generation"); + let (_root, verified) = notes + .verify_proof(proof.as_slice(), platform_version()) + .expect("proof verification"); + let verified_bodies: Vec> = verified + .iter() + .map(|document| payload_bytes(document, "body")) + .collect(); + assert_eq!(verified_bodies, bodies); + + let reactions = query_ordered( + &contract, + "reaction", + vec![ + equal("postId", Value::Identifier(POST)), + WhereClause { + field: "kind".to_string(), + operator: WhereOperator::In, + value: Value::Array(vec![Value::U64(7), Value::U64(3)]), + }, + ], + vec![("kind", true)], + Some(10), + ); + let outcome = drive + .query_documents(reactions.clone(), None, false, None, None) + .expect("in on a leading component executes"); + let kinds_and_owners: Vec<(u64, [u8; 32])> = outcome + .documents() + .iter() + .map(|document| { + ( + document + .properties() + .get("kind") + .and_then(|value| value.to_integer::().ok()) + .expect("kind decoded"), + document.owner_id().to_buffer(), + ) + }) + .collect(); + assert_eq!(kinds_and_owners, vec![(3, OWNER_1), (7, OWNER_3)]); + let (proof, _) = reactions + .clone() + .execute_with_proof(&drive, None, None, platform_version()) + .expect("proof generation"); + let (_root, verified) = reactions + .verify_proof(proof.as_slice(), platform_version()) + .expect("proof verification"); + assert_eq!(verified.len(), 2); + + assert_grovedb_is_consistent(&drive); +} diff --git a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/mod.rs b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/mod.rs index a28ac2c257e..f3e0c989771 100644 --- a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/mod.rs +++ b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/mod.rs @@ -30,6 +30,7 @@ mod chained_query_e2e_tests; mod composite_query_e2e_tests; mod countable_e2e_tests; mod index_only_e2e_tests; +mod index_only_scalar_terminal_e2e_tests; mod noncounted_sibling_e2e_tests; mod preallocated_index_e2e_tests; mod prefix_ranked_index_e2e_tests; diff --git a/packages/rs-drive/src/drive/document/delete/remove_indices_for_top_index_level_for_contract_operations/v2/mod.rs b/packages/rs-drive/src/drive/document/delete/remove_indices_for_top_index_level_for_contract_operations/v2/mod.rs index 0f2f177135c..41c28cfdec3 100644 --- a/packages/rs-drive/src/drive/document/delete/remove_indices_for_top_index_level_for_contract_operations/v2/mod.rs +++ b/packages/rs-drive/src/drive/document/delete/remove_indices_for_top_index_level_for_contract_operations/v2/mod.rs @@ -27,6 +27,7 @@ use crate::fees::op::LowLevelDriveOperation; use dpp::data_contract::accessors::v0::DataContractV0Getters; use dpp::data_contract::config::v0::DataContractConfigGettersV0; use dpp::data_contract::document_type::accessors::{DocumentTypeV0Getters, DocumentTypeV2Getters}; +use dpp::data_contract::document_type::is_flat_level_key; use crate::drive::document::paths::contract_document_type_path_vec; use dpp::version::PlatformVersion; @@ -91,14 +92,26 @@ impl Drive { let sub_level_index_count = index_level.sub_levels().len() as u32; if let Some(estimated_costs_only_with_layer_info) = estimated_costs_only_with_layer_info { - // On this level we will have a 0 and all the top index paths + // On this level we will have a 0 and all the top index paths. + // Property-name keys keep the historical 32-byte estimate; a + // FLAT indexOnly level is keyed by its zero-joined component + // names, which can be wider, and every entry write rewrites + // that node at its real key length. + let sub_level_key_max_size = index_level + .sub_levels() + .keys() + .filter(|name| is_flat_level_key(name)) + .map(|name| u8::try_from(name.len()).unwrap_or(u8::MAX)) + .max() + .unwrap_or(DEFAULT_HASH_SIZE_U8) + .max(DEFAULT_HASH_SIZE_U8); estimated_costs_only_with_layer_info.insert( KeyInfoPath::from_known_owned_path(contract_document_type_path.clone()), EstimatedLayerInformation { tree_type: TreeType::NormalTree, estimated_layer_count: ApproximateElements(sub_level_index_count + 1), estimated_layer_sizes: AllSubtrees( - DEFAULT_HASH_SIZE_U8, + sub_level_key_max_size, NoSumTrees, storage_flags.map(|s| s.serialized_size()), ), @@ -108,6 +121,60 @@ impl Drive { // next we need to store a reference to the document for each index for (name, sub_level) in index_level.sub_levels() { + // A FLAT indexOnly index: no property-name tree and no value + // level. Its level tree (created at registration, like a + // property-name tree) holds the `0` member bucket directly, so + // the entry is handled by the terminal branch straight below + // the level: `[…doctype, , 0, ]`. + if is_flat_level_key(name) { + let Some(index_type) = sub_level.has_index_with_type() else { + continue; + }; + let mut flat_path: Vec> = contract_document_type_path.clone(); + flat_path.push(Vec::from(name.as_bytes())); + if let Some(estimated_costs_only_with_layer_info) = + estimated_costs_only_with_layer_info + { + estimated_costs_only_with_layer_info.insert( + KeyInfoPath::from_known_owned_path(flat_path.clone()), + EstimatedLayerInformation { + tree_type: TreeType::NormalTree, + estimated_layer_count: ApproximateElements(1), + estimated_layer_sizes: AllSubtrees( + 1, + NoSumTrees, + storage_flags.map(|s| s.serialized_size()), + ), + }, + ); + } + let flat_path_info = if document_and_contract_info + .owned_document_info + .document_info + .is_document_size() + { + PathInfo::PathWithSizes(KeyInfoPath::from_known_owned_path(flat_path)) + } else { + PathInfo::PathAsVec::<0>(flat_path) + }; + self.remove_reference_for_index_level_for_contract_operations( + document_and_contract_info, + flat_path_info, + index_type, + false, + false, + &storage_flags, + previous_batch_operations, + estimated_costs_only_with_layer_info, + false, + event_id, + transaction, + batch_operations, + platform_version, + )?; + continue; + } + // The delete walker writes nothing itself, but its // estimation layers must describe the tree the insert path // actually laid down — including the meta-schema-v3 ranked diff --git a/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v0/mod.rs b/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v0/mod.rs index b057e4bb4c5..825a826c659 100644 --- a/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v0/mod.rs +++ b/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v0/mod.rs @@ -13,9 +13,9 @@ use std::collections::HashMap; use crate::drive::constants::CONTRACT_DOCUMENTS_PATH_HEIGHT; use crate::drive::document::document_reference_size; use crate::drive::document::index_level_tree_types::terminal_member_tree_type; -use crate::error::drive::DriveError; +use crate::drive::document::index_only::{index_only_member_key, index_only_terminal_max_key_size}; +use crate::drive::document::index_only_entry_payload_max_size; use crate::util::storage_flags::StorageFlags; -use dpp::document::document_methods::DocumentMethodsV0; use crate::drive::Drive; use crate::util::object_size_info::{DocumentAndContractInfo, DocumentInfoV0Methods, PathInfo}; @@ -65,10 +65,17 @@ impl Drive { // `add_reference_for_index_level_for_contract_operations_v0`. // `terminal` can only be `Some` on a PV14+ indexOnly contract, so // this branch is unreachable for every historical document. - if let Some(terminal_property) = index_type.terminal.as_deref() { + if let Some(terminal) = index_type.terminal.as_deref() { key_info_path.push(KnownKey(vec![0])); let member_tree_type = terminal_member_tree_type(index_type); + // The member key's estimated width follows the terminal property, + // mirroring the insert side. + let member_key_max_size = + index_only_terminal_max_key_size(document_type, terminal, platform_version)?; + // The stored item: the commitment plus the type's entry payload. + let entry_value_size = crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE + + index_only_entry_payload_max_size(document_type, platform_version)?; // Sum-bearing entries (`ItemWithSumItem`) carry the i64 sum // item alongside the commitment payload; mirror the insert @@ -77,9 +84,9 @@ impl Drive { // and propagates the subtraction — same as stored-type // `ReferenceWithSumItem` entries below. let estimated_value_size = if index_type.summable.is_some() { - crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE + 10 + entry_value_size + 10 } else { - crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE + entry_value_size }; if let Some(estimated_costs_only_with_layer_info) = estimated_costs_only_with_layer_info @@ -90,7 +97,7 @@ impl Drive { tree_type: member_tree_type, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems( - DEFAULT_HASH_SIZE_U8, + member_key_max_size, estimated_value_size, storage_flags.map(|s| s.serialized_size()), ), @@ -106,23 +113,19 @@ impl Drive { .document_info .get_borrowed_document_and_storage_flags() { - Some((document, _)) => document - .get_raw_for_document_type( - terminal_property, - document_type, - document_and_contract_info.owned_document_info.owner_id, - platform_version, - )? - .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( - "indexOnly terminal value must be present on delete: the \ - delete transition carries every property and the owner", - )))?, + Some((document, _)) => index_only_member_key( + document, + document_type, + terminal, + document_and_contract_info.owned_document_info.owner_id, + platform_version, + )?, None => event_id.to_vec(), }; let delete_apply_type = Self::stateless_delete_of_non_tree_for_costs( AllItems( - DEFAULT_HASH_SIZE_U8, + member_key_max_size, estimated_value_size, storage_flags.map(|s| s.serialized_size()), ), diff --git a/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v1/mod.rs b/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v1/mod.rs index 5f5b2ea2447..86e0c38e402 100644 --- a/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v1/mod.rs +++ b/packages/rs-drive/src/drive/document/delete/remove_reference_for_index_level_for_contract_operations/v1/mod.rs @@ -13,9 +13,10 @@ use std::collections::HashMap; use crate::drive::constants::CONTRACT_DOCUMENTS_PATH_HEIGHT; use crate::drive::document::document_reference_size; use crate::drive::document::index_level_tree_types::terminal_member_tree_type; +use crate::drive::document::index_only::{index_only_member_key, index_only_terminal_max_key_size}; +use crate::drive::document::index_only_entry_payload_max_size; use crate::error::drive::DriveError; use crate::util::storage_flags::StorageFlags; -use dpp::document::document_methods::DocumentMethodsV0; use crate::drive::Drive; use crate::util::object_size_info::{DocumentAndContractInfo, DocumentInfoV0Methods, PathInfo}; @@ -124,10 +125,17 @@ impl Drive { // `add_reference_for_index_level_for_contract_operations_v0`. // `terminal` can only be `Some` on a PV14+ indexOnly contract, so // this branch is unreachable for every historical document. - if let Some(terminal_property) = index_type.terminal.as_deref() { + if let Some(terminal) = index_type.terminal.as_deref() { key_info_path.push(KnownKey(vec![0])); let member_tree_type = terminal_member_tree_type(index_type); + // The member key's estimated width follows the terminal property, + // mirroring the insert side. + let member_key_max_size = + index_only_terminal_max_key_size(document_type, terminal, platform_version)?; + // The stored item: the commitment plus the type's entry payload. + let entry_value_size = crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE + + index_only_entry_payload_max_size(document_type, platform_version)?; // Sum-bearing entries (`ItemWithSumItem`) carry the i64 sum // item alongside the commitment payload; mirror the insert @@ -136,9 +144,9 @@ impl Drive { // and propagates the subtraction — same as stored-type // `ReferenceWithSumItem` entries below. let estimated_value_size = if index_type.summable.is_some() { - crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE + 10 + entry_value_size + 10 } else { - crate::drive::document::INDEX_ONLY_ROW_COMMITMENT_SIZE + entry_value_size }; if let Some(estimated_costs_only_with_layer_info) = estimated_costs_only_with_layer_info @@ -149,7 +157,7 @@ impl Drive { tree_type: member_tree_type, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems( - DEFAULT_HASH_SIZE_U8, + member_key_max_size, estimated_value_size, storage_flags.map(|s| s.serialized_size()), ), @@ -165,23 +173,19 @@ impl Drive { .document_info .get_borrowed_document_and_storage_flags() { - Some((document, _)) => document - .get_raw_for_document_type( - terminal_property, - document_type, - document_and_contract_info.owned_document_info.owner_id, - platform_version, - )? - .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( - "indexOnly terminal value must be present on delete: the \ - delete transition carries every property and the owner", - )))?, + Some((document, _)) => index_only_member_key( + document, + document_type, + terminal, + document_and_contract_info.owned_document_info.owner_id, + platform_version, + )?, None => event_id.to_vec(), }; let delete_apply_type = Self::stateless_delete_of_non_tree_for_costs( AllItems( - DEFAULT_HASH_SIZE_U8, + member_key_max_size, estimated_value_size, storage_flags.map(|s| s.serialized_size()), ), @@ -202,7 +206,12 @@ impl Drive { // keeps every tree, so a later entry costs the same as this one // did. Trees a pre-flag fallback created are kept the same way // — if-not-exists inserts make the two histories converge. - let stop_path_height = if index_type.preallocated { + // A FLAT index's entries sit at `[…doctype, , 0]`: + // the flat level tree is registration-time structure (like a + // property-name tree), so the climb stops at its `0` bucket + // exactly as it does on a preallocated index. The level info + // carries the flag from the index that terminates there. + let stop_path_height = if index_type.preallocated || index_type.flat { u16::try_from(key_info_path.len() - 1).map_err(|_| { Error::Drive(DriveError::CorruptedCodeExecution( "index path height must fit in u16", diff --git a/packages/rs-drive/src/drive/document/index_level_tree_types.rs b/packages/rs-drive/src/drive/document/index_level_tree_types.rs index 05860ddd970..2245319577f 100644 --- a/packages/rs-drive/src/drive/document/index_level_tree_types.rs +++ b/packages/rs-drive/src/drive/document/index_level_tree_types.rs @@ -339,6 +339,7 @@ mod tests { ranked_averageable: false, terminal: None, preallocated: false, + flat: false, } } diff --git a/packages/rs-drive/src/drive/document/index_only.rs b/packages/rs-drive/src/drive/document/index_only.rs index f848c91b1d5..269afe34059 100644 --- a/packages/rs-drive/src/drive/document/index_only.rs +++ b/packages/rs-drive/src/drive/document/index_only.rs @@ -22,16 +22,18 @@ use crate::drive::constants::CONTRACT_DOCUMENTS_PATH_HEIGHT; use crate::drive::document::index_level_tree_types::terminal_member_tree_type; +use crate::drive::document::index_only_item_estimated_value_size; use crate::drive::document::time_range_ttl::entry_key_bucket_start; -use crate::drive::document::INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE; use crate::drive::Drive; use crate::error::drive::DriveError; use crate::error::Error; use crate::fees::op::LowLevelDriveOperation; use crate::util::grove_operations::{DirectQueryType, QueryTarget}; +use crate::util::type_constants::DEFAULT_HASH_SIZE_U8; use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; use dpp::data_contract::document_type::{DocumentTypeRef, Index}; use dpp::document::document_methods::DocumentMethodsV0; +use dpp::document::property_names::OWNER_ID; use dpp::document::{Document, DocumentV0Getters}; use dpp::identifier::Identifier; use dpp::version::PlatformVersion; @@ -168,19 +170,33 @@ impl Drive { }) .collect(); } + // A flat index (no prefix properties) keeps its entries under the + // level keyed by its terminal's names, between the doctype and the + // `0` bucket. + if let Some(flat_key) = index.flat_level_key() { + for path in paths.iter_mut() { + path.push(flat_key.as_bytes().to_vec()); + } + } for path in paths.iter_mut() { path.push(vec![0]); } - let terminal = - index - .terminal - .as_deref() - .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( - "index_only_entry_paths_and_key requires an indexOnly index (terminal is \ + if index.terminal.is_none() { + return Err(Error::Drive(DriveError::CorruptedCodeExecution( + "index_only_entry_paths_and_key requires an indexOnly index (terminal is \ always Some there after parse normalization)", - )))?; - let member_key = raw_value_for(terminal)?; + ))); + } + // The member key: the terminal components' encoded values, + // concatenated, exactly as the write side keys the entry. + let member_key = index_only_member_key( + document, + document_type, + index.terminal_components(), + owner_id, + platform_version, + )?; Ok((paths, member_key)) } @@ -244,8 +260,11 @@ impl Drive { &platform_version.drive, )?; let matches = match element { + // The commitment is the item's first 32 bytes; an entry + // payload (the type's value slot) follows it and is covered + // by the commitment, so the prefix is what binds. Some(grovedb::Element::Item(payload, _)) => { - payload == expected_commitment.as_slice() + payload.get(..expected_commitment.len()) == Some(expected_commitment.as_slice()) } // Summable indexes store `ItemWithSumItem(commitment, // amount)`; the commitment payload plays the same binding @@ -253,7 +272,7 @@ impl Drive { // of the document's properties, so it is already covered // by the commitment. Some(grovedb::Element::ItemWithSumItem(payload, _, _)) => { - payload == expected_commitment.as_slice() + payload.get(..expected_commitment.len()) == Some(expected_commitment.as_slice()) } _ => false, }; @@ -283,6 +302,8 @@ impl Drive { drive_operations: &mut Vec, platform_version: &PlatformVersion, ) -> Result<(), Error> { + let estimated_item_value_size = + index_only_item_estimated_value_size(document_type, platform_version)?; for index in document_type.indexes().values() { let (paths, member_key) = Self::index_only_entry_paths_and_key( contract_id, @@ -295,6 +316,16 @@ impl Drive { // keys the paths are built from, so the claimed tree type is // the one the walkers created the `0` member bucket with. let mut level = document_type.index_structure(); + // A flat index terminates at its own level, directly under + // the root. + if let Some(flat_key) = index.flat_level_key() { + level = level.sub_levels().get(&flat_key).ok_or(Error::Drive( + DriveError::CorruptedCodeExecution( + "a flat indexOnly index resolves to its flat level: the structure is \ + built from the same indexes", + ), + ))?; + } for (position, property) in index.properties.iter().enumerate() { let level_key = index.level_key(position, &property.name); level = level.sub_levels().get(&*level_key).ok_or(Error::Drive( @@ -311,9 +342,9 @@ impl Drive { ))?; let in_tree_type = terminal_member_tree_type(type_info); let estimated_value_size = if type_info.summable.is_some() { - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE + 10 + estimated_item_value_size + 10 } else { - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE + estimated_item_value_size }; for path in paths { let path_refs: Vec<&[u8]> = path.iter().map(|segment| segment.as_slice()).collect(); @@ -371,3 +402,73 @@ impl Drive { Ok(false) } } + +/// The widest member key an indexOnly index's terminal can produce: the +/// sum over its components of 32 bytes for `$ownerId` and, otherwise, the +/// component property's maximum encoded size (a bounded byte array or +/// string encodes to at most its declared bound — the string bound counts +/// characters of up to four bytes — and every other indexable type +/// encodes to a fixed width). Fee estimation sizes the `0` member bucket's +/// keys by it, so the dry run upper-bounds the applied fee for a wide or +/// composite terminal exactly as it does for a 32-byte one. The contract +/// parser keeps every terminal within grovedb's 255-byte key cap, so the +/// clamp below is a guard, not a path. +pub(crate) fn index_only_terminal_max_key_size( + document_type: DocumentTypeRef, + terminal: &[String], + platform_version: &PlatformVersion, +) -> Result { + let mut total: u32 = 0; + for component in terminal { + let width: u32 = if component == OWNER_ID { + u32::from(DEFAULT_HASH_SIZE_U8) + } else { + let property = + document_type + .flattened_properties() + .get(component) + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "indexOnly terminal names a property the document type lacks: the \ + contract parser admits only $ownerId or schema properties as terminal \ + components", + )))?; + u32::from( + property + .property_type + .max_byte_size(platform_version) + .map_err(|e| Error::Protocol(Box::new(e)))? + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "indexOnly terminal component has an unbounded type: the contract \ + parser refuses arrays and objects as terminal components", + )))?, + ) + }; + total = total.saturating_add(width); + } + Ok(u8::try_from(total).unwrap_or(u8::MAX)) +} + +/// The member key `document` produces under a terminal: the components' +/// values in their tree-key encoding, concatenated in the terminal's +/// order — one component for a plain terminal, several for a composite +/// one. The write path's twin of the query side's +/// `serialize_value_for_key` concatenation. +pub(crate) fn index_only_member_key( + document: &Document, + document_type: DocumentTypeRef, + terminal: &[String], + owner_id: Option<[u8; 32]>, + platform_version: &PlatformVersion, +) -> Result, Error> { + let mut member_key = Vec::new(); + for component in terminal { + let encoded = document + .get_raw_for_document_type(component, document_type, owner_id, platform_version)? + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "indexOnly terminal value must be present: the parser requires every \ + indexOnly property (and $ownerId) to be set", + )))?; + member_key.extend(encoded); + } + Ok(member_key) +} diff --git a/packages/rs-drive/src/drive/document/index_only_entry_payload.rs b/packages/rs-drive/src/drive/document/index_only_entry_payload.rs new file mode 100644 index 00000000000..ced0310c26f --- /dev/null +++ b/packages/rs-drive/src/drive/document/index_only_entry_payload.rs @@ -0,0 +1,326 @@ +//! The **entry payload** of an indexOnly document type: the value slot. +//! +//! A document type may list top-level properties under `entryPayload`. +//! They sit in no index; instead every entry's item carries them after the +//! 32-byte row commitment, so an entry reads `Item(commitment ‖ payload)` +//! (or `ItemWithSumItem(commitment ‖ payload, amount)` under a summable +//! index). The commitment still hashes them, so the delete probes and the +//! executed-transition verifier keep comparing the first 32 bytes only, +//! and the payload is recovered by decoding the proved element. +//! +//! Layout: for every payload property in property-name order (the order +//! the parsed `BTreeSet` iterates), a big-endian `u16` length followed by +//! the value's bytes: raw bytes for byte arrays, UTF-8 for strings, and +//! tree-key encoding for other scalars. Length framing preserves empty +//! values without the tree-key encoding's empty/null sentinels. The parser +//! bounds every payload property and caps their sum, which is what lets +//! the length frame be two bytes and fee estimation size the entry value +//! by the bounds. + +use crate::drive::document::INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE; +use crate::error::drive::DriveError; +use crate::error::Error; +use dpp::data_contract::document_type::accessors::{DocumentTypeV0Getters, DocumentTypeV2Getters}; +use dpp::data_contract::document_type::{DocumentPropertyType, DocumentTypeRef}; +use dpp::document::{Document, DocumentV0Getters}; +use dpp::platform_value::Value; +use dpp::version::PlatformVersion; +use std::collections::BTreeMap; + +/// The bytes of length frame each payload property carries. +pub const INDEX_ONLY_ENTRY_PAYLOAD_LENGTH_FRAME: u32 = 2; + +/// The most bytes `document_type`'s entry payload can encode to: the sum +/// of every payload property's declared bound plus its length frame. Zero +/// on a type without an entry payload. The parser guarantees every payload +/// property is bounded, so an unbounded one here is a corrupted type. +pub fn index_only_entry_payload_max_size( + document_type: DocumentTypeRef, + platform_version: &PlatformVersion, +) -> Result { + let mut total: u32 = 0; + for property_name in document_type.entry_payload() { + let property = document_type + .properties() + .get(property_name) + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "an entryPayload property must be a top-level property of its document type: \ + the contract parser enforces it", + )))?; + let max_width = property + .property_type + .max_byte_size(platform_version) + .map_err(|e| Error::Protocol(Box::new(e)))? + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "an entryPayload property must be bounded: the contract parser enforces it", + )))?; + total = total + .saturating_add(u32::from(max_width)) + .saturating_add(INDEX_ONLY_ENTRY_PAYLOAD_LENGTH_FRAME); + } + Ok(total) +} + +/// The value size fee estimation claims for one of `document_type`'s entry +/// items: the padded commitment estimate plus the payload bound. Every +/// estimating site (the entry-insert terminal, the preallocated-tree +/// estimate, the delete walkers and the delete-side probe estimate) sizes +/// the item through this one function so they cannot drift. +pub fn index_only_item_estimated_value_size( + document_type: DocumentTypeRef, + platform_version: &PlatformVersion, +) -> Result { + Ok( + INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE.saturating_add(index_only_entry_payload_max_size( + document_type, + platform_version, + )?), + ) +} + +/// The bytes one entry payload value contributes: raw bytes for a byte +/// array, UTF-8 for a string, and the tree-key encoding's fixed +/// order-preserving widths for the numeric kinds. Unlike a key, a payload +/// value is not capped at 255 bytes — the parser bounds it by the field +/// value limit instead. Also what the row commitment hashes for a payload +/// property, so the commitment and the stored value agree byte for byte. +pub fn encode_index_only_entry_payload_value( + property_type: &DocumentPropertyType, + value: &Value, +) -> Result, Error> { + // Payload lengths already distinguish empty values; the tree-key + // string sentinel would conflate "" and "\0" in both the stored + // payload and its row commitment. + if let (DocumentPropertyType::String(_), Value::Text(text)) = (property_type, value) { + return Ok(text.as_bytes().to_vec()); + } + property_type + .encode_value_for_tree_keys(value) + .map_err(|e| Error::Protocol(Box::new(e))) +} + +/// Encode `document`'s entry payload — every `entryPayload` property in +/// name order, length-framed in its payload encoding. Empty on a type +/// without an entry payload. A missing payload property is a corrupted +/// document: the parser requires every payload property. +pub fn encode_index_only_entry_payload( + document: &Document, + document_type: DocumentTypeRef, +) -> Result, Error> { + let mut payload = Vec::new(); + for property_name in document_type.entry_payload() { + let property = document_type + .properties() + .get(property_name) + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "an entryPayload property must be a top-level property of its document type", + )))?; + let value = document + .properties() + .get(property_name) + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "an indexOnly document must carry every entryPayload property: the parser \ + requires them", + )))?; + let encoded = encode_index_only_entry_payload_value(&property.property_type, value)?; + let length = u16::try_from(encoded.len()).map_err(|_| { + Error::Drive(DriveError::CorruptedCodeExecution( + "an entryPayload value exceeds its bound: schema validation admits no such \ + document", + )) + })?; + payload.extend_from_slice(&length.to_be_bytes()); + payload.extend_from_slice(&encoded); + } + Ok(payload) +} + +/// Decode an entry item's payload (the bytes after the 32-byte row +/// commitment) back into `document_type`'s entry payload properties. Fails +/// closed on any framing mismatch: a truncated, overlong or misframed +/// payload is a corrupted entry, never a partial document. +pub fn decode_index_only_entry_payload( + document_type: DocumentTypeRef, + payload: &[u8], +) -> Result, Error> { + let corrupted = + |message: &'static str| Error::Drive(DriveError::CorruptedCodeExecution(message)); + let mut properties = BTreeMap::new(); + let mut cursor = 0usize; + for property_name in document_type.entry_payload() { + let property = document_type + .properties() + .get(property_name) + .ok_or(corrupted( + "an entryPayload property must be a top-level property of its document type", + ))?; + let frame = payload + .get(cursor..cursor + INDEX_ONLY_ENTRY_PAYLOAD_LENGTH_FRAME as usize) + .ok_or(corrupted( + "indexOnly entry payload is truncated before a property's length frame", + ))?; + let length = usize::from(u16::from_be_bytes([frame[0], frame[1]])); + cursor += INDEX_ONLY_ENTRY_PAYLOAD_LENGTH_FRAME as usize; + let encoded = payload.get(cursor..cursor + length).ok_or(corrupted( + "indexOnly entry payload is truncated inside a property", + ))?; + cursor += length; + let value = match &property.property_type { + // These are required values, so an empty frame is an empty + // byte array or string, never the tree-key null sentinel. + DocumentPropertyType::ByteArray(_) => Value::Bytes(encoded.to_vec()), + DocumentPropertyType::String(_) => Value::Text( + String::from_utf8(encoded.to_vec()) + .map_err(|_| corrupted("indexOnly entry payload contains invalid UTF-8"))?, + ), + // Every other payload type encodes to a fixed width, so an + // empty frame is a corrupted entry; the tree-key decoder would + // read it as the null sentinel instead. + _ if encoded.is_empty() => { + return Err(corrupted( + "indexOnly entry payload carries an empty frame for a fixed-width property", + )); + } + property_type => property_type + .decode_value_for_tree_keys(encoded) + .map_err(|e| Error::Protocol(Box::new(e)))?, + }; + properties.insert(property_name.clone(), value); + } + if cursor != payload.len() { + return Err(corrupted( + "indexOnly entry payload carries bytes past its last property", + )); + } + Ok(properties) +} + +#[cfg(test)] +mod tests { + use super::*; + use dpp::data_contract::config::DataContractConfig; + use dpp::data_contract::document_type::random_document::CreateRandomDocument; + use dpp::data_contract::document_type::DocumentType; + use dpp::platform_value::platform_value; + use dpp::platform_value::Identifier; + use std::collections::BTreeMap; + + /// An indexOnly type whose value slot holds a fixed-width integer and + /// a bounded string, in that (name) order. + fn payload_type() -> DocumentType { + let platform_version = PlatformVersion::latest(); + let config = DataContractConfig::default_for_version(platform_version).expect("config"); + let schema = platform_value!({ + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [{ "name": "byOwner", "terminal": "$ownerId" }], + "entryPayload": ["count", "text"], + "properties": { + "count": { "type": "integer", "position": 0 }, + "text": { "type": "string", "maxLength": 8, "position": 1 } + }, + "required": ["count", "text"], + "additionalProperties": false + }); + DocumentType::try_from_schema( + Identifier::random(), + 1, + config.version(), + "entry", + schema, + None, + &BTreeMap::new(), + &config, + false, + &mut Vec::new(), + platform_version, + ) + .expect("the payload type parses") + } + + fn frame(bytes: &[u8]) -> Vec { + let mut framed = (bytes.len() as u16).to_be_bytes().to_vec(); + framed.extend_from_slice(bytes); + framed + } + + fn count_frame(value: i64) -> Vec { + frame( + &DocumentPropertyType::I64 + .encode_value_for_tree_keys(&Value::I64(value)) + .expect("i64 key"), + ) + } + + fn decode_error(payload: &[u8]) -> String { + decode_index_only_entry_payload(payload_type().as_ref(), payload) + .expect_err("a malformed payload must be refused") + .to_string() + } + + #[test] + fn should_round_trip_a_documents_payload() { + let document_type = payload_type(); + let platform_version = PlatformVersion::latest(); + let document = document_type + .random_document(Some(7), platform_version) + .expect("random document"); + let encoded = encode_index_only_entry_payload(&document, document_type.as_ref()) + .expect("payload encodes"); + let decoded = decode_index_only_entry_payload(document_type.as_ref(), &encoded) + .expect("payload decodes"); + for name in ["count", "text"] { + assert_eq!(decoded.get(name), document.properties().get(name), "{name}"); + } + assert_eq!(decoded.len(), 2); + } + + #[test] + fn should_decode_a_well_framed_payload() { + let mut payload = count_frame(-5); + payload.extend(frame(b"abc")); + let decoded = + decode_index_only_entry_payload(payload_type().as_ref(), &payload).expect("decodes"); + assert_eq!(decoded.get("count"), Some(&Value::I64(-5))); + assert_eq!(decoded.get("text"), Some(&Value::Text("abc".to_string()))); + } + + #[test] + fn should_refuse_a_truncated_length_frame() { + let mut payload = count_frame(1); + payload.push(0); + assert!(decode_error(&payload).contains("truncated before a property's length frame")); + } + + #[test] + fn should_refuse_a_frame_longer_than_the_payload() { + let mut payload = count_frame(1); + payload.extend(frame(b"abc")); + payload.pop(); + assert!(decode_error(&payload).contains("truncated inside a property")); + } + + #[test] + fn should_refuse_bytes_past_the_last_property() { + let mut payload = count_frame(1); + payload.extend(frame(b"abc")); + payload.push(0); + assert!(decode_error(&payload).contains("past its last property")); + } + + #[test] + fn should_refuse_invalid_utf8_in_a_string_payload() { + let mut payload = count_frame(1); + payload.extend(frame(&[0xFF, 0xFE])); + assert!(decode_error(&payload).contains("invalid UTF-8")); + } + + #[test] + fn should_refuse_an_empty_frame_for_a_fixed_width_property() { + let mut payload = frame(&[]); + payload.extend(frame(b"abc")); + assert!(decode_error(&payload).contains("empty frame for a fixed-width property")); + } +} diff --git a/packages/rs-drive/src/drive/document/index_only_row_commitment.rs b/packages/rs-drive/src/drive/document/index_only_row_commitment.rs index b302e2860dd..0bbe0cec0d2 100644 --- a/packages/rs-drive/src/drive/document/index_only_row_commitment.rs +++ b/packages/rs-drive/src/drive/document/index_only_row_commitment.rs @@ -2,12 +2,14 @@ //! item stores, binding the independently stored index projections of one //! document back into one logical row. +#[cfg(any(feature = "server", feature = "verify"))] +use crate::drive::document::encode_index_only_entry_payload_value; #[cfg(any(feature = "server", feature = "verify"))] use crate::error::drive::DriveError; #[cfg(any(feature = "server", feature = "verify"))] use crate::error::Error; #[cfg(any(feature = "server", feature = "verify"))] -use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; +use dpp::data_contract::document_type::accessors::{DocumentTypeV0Getters, DocumentTypeV2Getters}; #[cfg(any(feature = "server", feature = "verify"))] use dpp::data_contract::document_type::{DocumentPropertyType, DocumentTypeRef}; #[cfg(any(feature = "server", feature = "verify"))] @@ -99,13 +101,37 @@ pub fn index_only_row_commitment_with_preimage_size( property_names.sort(); for property_name in property_names { - let Some(raw) = document.get_raw_for_document_type( - property_name, - document_type, - owner_id, - platform_version, - )? - else { + // An entry payload property is a value, not a key: it is committed + // through the payload encoding, which carries no 255-byte key cap + // (the parser bounds it by the field value limit instead). + let raw = if document_type + .entry_payload() + .contains(property_name.as_str()) + { + match document.properties().get(property_name.as_str()) { + Some(value) => { + let property = document_type + .flattened_properties() + .get(property_name) + .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( + "an entryPayload property must be a property of its document type", + )))?; + Some(encode_index_only_entry_payload_value( + &property.property_type, + value, + )?) + } + None => None, + } + } else { + document.get_raw_for_document_type( + property_name, + document_type, + owner_id, + platform_version, + )? + }; + let Some(raw) = raw else { if document_type .required_fields() .contains(property_name.as_str()) diff --git a/packages/rs-drive/src/drive/document/insert/add_indices_for_top_index_level_for_contract_operations/v2/mod.rs b/packages/rs-drive/src/drive/document/insert/add_indices_for_top_index_level_for_contract_operations/v2/mod.rs index c04227e2ef9..465b5513c4f 100644 --- a/packages/rs-drive/src/drive/document/insert/add_indices_for_top_index_level_for_contract_operations/v2/mod.rs +++ b/packages/rs-drive/src/drive/document/insert/add_indices_for_top_index_level_for_contract_operations/v2/mod.rs @@ -14,6 +14,7 @@ use crate::fees::op::LowLevelDriveOperation; use dpp::data_contract::accessors::v0::DataContractV0Getters; use dpp::data_contract::config::v0::DataContractConfigGettersV0; use dpp::data_contract::document_type::accessors::{DocumentTypeV0Getters, DocumentTypeV2Getters}; +use dpp::data_contract::document_type::is_flat_level_key; use dpp::version::PlatformVersion; @@ -94,14 +95,26 @@ impl Drive { let sub_level_index_count = index_level.sub_levels().len() as u32; if let Some(estimated_costs_only_with_layer_info) = estimated_costs_only_with_layer_info { - // On this level we will have a 0 and all the top index paths + // On this level we will have a 0 and all the top index paths. + // Property-name keys keep the historical 32-byte estimate; a + // FLAT indexOnly level is keyed by its zero-joined component + // names, which can be wider, and every entry write rewrites + // that node at its real key length. + let sub_level_key_max_size = index_level + .sub_levels() + .keys() + .filter(|name| is_flat_level_key(name)) + .map(|name| u8::try_from(name.len()).unwrap_or(u8::MAX)) + .max() + .unwrap_or(DEFAULT_HASH_SIZE_U8) + .max(DEFAULT_HASH_SIZE_U8); estimated_costs_only_with_layer_info.insert( KeyInfoPath::from_known_owned_path(contract_document_type_path.clone()), EstimatedLayerInformation { tree_type: TreeType::NormalTree, estimated_layer_count: ApproximateElements(sub_level_index_count + 1), estimated_layer_sizes: AllSubtrees( - DEFAULT_HASH_SIZE_U8, + sub_level_key_max_size, NoSumTrees, storage_flags.map(|s| s.serialized_size()), ), @@ -111,6 +124,58 @@ impl Drive { // next we need to store a reference to the document for each index for (name, sub_level) in index_level.sub_levels() { + // A FLAT indexOnly index: no property-name tree and no value + // level. Its level tree (created at registration, like a + // property-name tree) holds the `0` member bucket directly, so + // the entry is handled by the terminal branch straight below + // the level: `[…doctype, , 0, ]`. + if is_flat_level_key(name) { + let Some(index_type) = sub_level.has_index_with_type() else { + continue; + }; + let mut flat_path: Vec> = contract_document_type_path.clone(); + flat_path.push(Vec::from(name.as_bytes())); + if let Some(estimated_costs_only_with_layer_info) = + estimated_costs_only_with_layer_info + { + estimated_costs_only_with_layer_info.insert( + KeyInfoPath::from_known_owned_path(flat_path.clone()), + EstimatedLayerInformation { + tree_type: TreeType::NormalTree, + estimated_layer_count: ApproximateElements(1), + estimated_layer_sizes: AllSubtrees( + 1, + NoSumTrees, + storage_flags.map(|s| s.serialized_size()), + ), + }, + ); + } + let flat_path_info = if document_and_contract_info + .owned_document_info + .document_info + .is_document_size() + { + PathInfo::PathWithSizes(KeyInfoPath::from_known_owned_path(flat_path)) + } else { + PathInfo::PathAsVec::<0>(flat_path) + }; + self.add_reference_for_index_level_for_contract_operations( + document_and_contract_info, + flat_path_info, + index_type, + false, + false, + previous_batch_operations, + &storage_flags, + estimated_costs_only_with_layer_info, + transaction, + batch_operations, + platform_version, + )?; + continue; + } + // The top-level property-name tree is created once, at // contract registration — this walker never writes it, so it // has no use for `tree_types.ranked_axes`. The resolved type diff --git a/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/mod.rs b/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/mod.rs index 086bd8255bb..50e75f259e7 100644 --- a/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/mod.rs +++ b/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/mod.rs @@ -41,9 +41,10 @@ use crate::drive::document::index_level_tree_types::{ index_level_tree_types_with_continuation_demotion, terminal_member_tree_type, terminal_value_tree_type, }; +use crate::drive::document::index_only::index_only_terminal_max_key_size; +use crate::drive::document::index_only_item_estimated_value_size; use crate::drive::document::paths::contract_document_type_path_vec; use crate::drive::document::unique_event_id; -use crate::drive::document::INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE; use crate::drive::Drive; use crate::error::drive::DriveError; use crate::error::fee::FeeError; @@ -66,6 +67,9 @@ use grovedb::EstimatedSumTrees::NoSumTrees; use grovedb::{EstimatedLayerInformation, TransactionArg, TreeType}; use std::collections::HashMap; +#[cfg(test)] +mod tests; + impl Drive { /// For every preallocated index (on any indexOnly document type of the /// contract) whose binding targets the document type being inserted, @@ -467,10 +471,21 @@ impl Drive { // Same per-entry padding (and sum-item worst case) the // entry-insert terminal claims for this layer — see // `add_index_only_terminal_item_operations`. + let referring_type = contract + .document_type_for_name(referring_type_name) + .map_err(|e| Error::Protocol(Box::new(dpp::ProtocolError::DataContractError(e))))?; + let estimated_item_value_size = + index_only_item_estimated_value_size(referring_type, platform_version)?; + let member_key_max_size = match level_info.terminal.as_deref() { + Some(terminal) => { + index_only_terminal_max_key_size(referring_type, terminal, platform_version)? + } + None => DEFAULT_HASH_SIZE_U8, + }; let estimated_value_size = if level_info.summable.is_some() { - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE + 10 + estimated_item_value_size + 10 } else { - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE + estimated_item_value_size }; estimated_costs_only_with_layer_info.insert( path_info.convert_to_key_info_path(), @@ -478,7 +493,7 @@ impl Drive { tree_type: member_tree_type, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems( - DEFAULT_HASH_SIZE_U8, + member_key_max_size, estimated_value_size, storage_flags.map(|s| s.serialized_size()), ), diff --git a/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/tests.rs b/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/tests.rs new file mode 100644 index 00000000000..5b8cfd589f6 --- /dev/null +++ b/packages/rs-drive/src/drive/document/insert/add_preallocated_index_tree_operations/tests.rs @@ -0,0 +1,73 @@ +use super::*; +use crate::util::object_size_info::DocumentInfo::DocumentRefInfo; +use crate::util::object_size_info::OwnedDocumentInfo; +use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; +use dpp::data_contract::document_type::random_document::CreateRandomDocument; +use dpp::document::DocumentV0Getters; +use dpp::prelude::DataContract; +use dpp::tests::json_document::json_document_to_json_value; +use serde_json::json; + +#[test] +fn should_estimate_composite_terminal_width_in_preallocated_member_layers() { + let platform_version = PlatformVersion::latest(); + let drive = setup_drive_with_initial_state_structure(None); + let mut schema = json_document_to_json_value( + "tests/supporting_files/contract/yappr-likes/yappr-likes-preallocated-contract.json", + ) + .expect("read contract fixture"); + let like = &mut schema["documentSchemas"]["like"]; + like["properties"]["nonce"] = json!({ + "type": "array", "byteArray": true, "minItems": 33, "maxItems": 33, "position": 2 + }); + like["required"] + .as_array_mut() + .expect("required fields") + .push(json!("nonce")); + for index in like["indices"].as_array_mut().expect("indices") { + if index["preallocated"] == true { + index["terminal"] = json!(["nonce", "$ownerId"]); + } + } + let contract = DataContract::try_from_platform_versioned( + serde_json::from_value(schema).expect("contract serialization format"), + false, + &mut vec![], + platform_version, + ) + .expect("parse composite-terminal contract"); + let post_type = contract.document_type_for_name("post").expect("post type"); + let post = post_type + .random_document(Some(1), platform_version) + .expect("post"); + let mut layers = Some(HashMap::new()); + drive + .add_preallocated_index_tree_operations_for_referring_types( + &DocumentAndContractInfo { + owned_document_info: OwnedDocumentInfo { + document_info: DocumentRefInfo((&post, None)), + owner_id: None, + }, + contract: &contract, + document_type: post_type, + }, + &mut None, + &mut layers, + None, + &mut vec![], + platform_version, + ) + .expect("estimate preallocation"); + + let mut member_path = contract_document_type_path_vec(contract.id_ref().as_bytes(), "like"); + member_path.extend([b"postId".to_vec(), post.id().to_vec(), vec![0]]); + let layers = layers.expect("estimated layers"); + let member_layer = layers + .get(&KeyInfoPath::from_known_owned_path(member_path)) + .expect("byPost member layer must be estimated"); + // A 33-byte nonce followed by the 32-byte owner identifier keys this bucket. + assert!( + matches!(member_layer.estimated_layer_sizes, AllItems(65, _, _)), + "the estimate must cover every terminal component: {member_layer:?}", + ); +} diff --git a/packages/rs-drive/src/drive/document/insert/add_reference_for_index_level_for_contract_operations/v0/mod.rs b/packages/rs-drive/src/drive/document/insert/add_reference_for_index_level_for_contract_operations/v0/mod.rs index 823088c98ee..e835bffb67f 100644 --- a/packages/rs-drive/src/drive/document/insert/add_reference_for_index_level_for_contract_operations/v0/mod.rs +++ b/packages/rs-drive/src/drive/document/insert/add_reference_for_index_level_for_contract_operations/v0/mod.rs @@ -1,10 +1,13 @@ use crate::drive::constants::STORAGE_FLAGS_SIZE; use crate::drive::document::index_level_tree_types::terminal_member_tree_type; -use crate::drive::document::INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE; +use crate::drive::document::index_only::{index_only_member_key, index_only_terminal_max_key_size}; use crate::drive::document::{ document_reference_size, make_document_reference, make_document_reference_with_sum_item, read_document_sum_contribution, }; +use crate::drive::document::{ + encode_index_only_entry_payload, index_only_item_estimated_value_size, +}; use crate::drive::Drive; use crate::error::drive::DriveError; use crate::error::Error; @@ -24,7 +27,6 @@ use crate::util::storage_flags::StorageFlags; use crate::util::type_constants::DEFAULT_HASH_SIZE_U8; use dpp::data_contract::document_type::methods::DocumentTypeBasicMethods; use dpp::data_contract::document_type::IndexLevelTypeInfo; -use dpp::document::document_methods::DocumentMethodsV0; use dpp::document::Document; use dpp::document::DocumentV0Getters; use dpp::version::PlatformVersion; @@ -69,12 +71,12 @@ impl Drive { // below meta-schema v3), so this branch is unreachable for every // historical document — the same in-place gating the count and sum // flags in this function already rely on. - if let Some(terminal_property) = index_type.terminal.as_deref() { + if let Some(terminal) = index_type.terminal.as_deref() { return self.add_index_only_terminal_item_operations( document_and_contract_info, index_path_info, index_type, - terminal_property, + terminal, previous_batch_operations, storage_flags, estimated_costs_only_with_layer_info, @@ -349,7 +351,8 @@ impl Drive { /// The indexOnly terminal: writes `[…index path, 0, ] → /// Item(commitment, flags)` — the member key is the terminal property's - /// value (`$ownerId` or a refersTo-typed identifier) sitting exactly + /// value in its tree-key encoding (`$ownerId` or any indexable schema + /// property, sized for estimation by its declared bound) sitting exactly /// where a normal non-unique index keys by document id, and the element /// payload is the row commitment because the entry IS the row. Storage /// flags ride the element flags for epoch/owner refunds, exactly as on @@ -378,7 +381,7 @@ impl Drive { document_and_contract_info: &DocumentAndContractInfo, mut index_path_info: PathInfo<0>, index_type: &IndexLevelTypeInfo, - terminal_property: &str, + terminal: &[String], previous_batch_operations: &mut Option<&mut Vec>, storage_flags: &Option<&StorageFlags>, estimated_costs_only_with_layer_info: &mut Option< @@ -392,6 +395,19 @@ impl Drive { let member_tree_type = terminal_member_tree_type(index_type); let sum_property_name: Option<&str> = index_type.summable.as_deref(); + // The member key's estimated width follows the terminal property: + // 32 bytes for `$ownerId`, the declared bound otherwise. + let member_key_max_size = index_only_terminal_max_key_size( + document_and_contract_info.document_type, + terminal, + platform_version, + )?; + // The item's estimated value: the padded commitment plus the type's + // entry payload bound, when it declares one. + let estimated_item_value_size = index_only_item_estimated_value_size( + document_and_contract_info.document_type, + platform_version, + )?; // The `0` storage-marker tree, byte-identical in position to the // non-unique layout above. @@ -448,20 +464,20 @@ impl Drive { // element envelope; 10 bytes is the worst case the sum-aware space // helpers reserve. let estimated_value_size = if sum_property_name.is_some() { - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE + 10 + estimated_item_value_size + 10 } else { - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE + estimated_item_value_size }; let item_space = if sum_property_name.is_some() { Element::required_item_with_sum_item_space( - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE, + estimated_item_value_size, STORAGE_FLAGS_SIZE, &drive_version.grove_version, )? } else { Element::required_item_space( - INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE, + estimated_item_value_size, STORAGE_FLAGS_SIZE, &drive_version.grove_version, )? @@ -474,7 +490,7 @@ impl Drive { tree_type: member_tree_type, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems( - DEFAULT_HASH_SIZE_U8, + member_key_max_size, estimated_value_size, storage_flags.map(|s| s.serialized_size()), ), @@ -482,27 +498,20 @@ impl Drive { ); } - // Member key: the terminal property's value — 32 bytes, since the - // parser only admits `$ownerId` or identifier-typed refersTo - // properties as terminals. + // Member key: the terminal property's value in its tree-key + // encoding — the same bytes a prefix level would key by. let member_key: Option> = match document_and_contract_info .owned_document_info .document_info .get_borrowed_document_and_storage_flags() { - Some((document, _)) => Some( - document - .get_raw_for_document_type( - terminal_property, - document_and_contract_info.document_type, - document_and_contract_info.owned_document_info.owner_id, - platform_version, - )? - .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( - "indexOnly terminal value must be present: the parser requires \ - every indexOnly property (and $ownerId) to be set", - )))?, - ), + Some((document, _)) => Some(index_only_member_key( + document, + document_and_contract_info.document_type, + terminal, + document_and_contract_info.owned_document_info.owner_id, + platform_version, + )?), // Estimation-only document info carries no values. None => None, }; @@ -514,7 +523,7 @@ impl Drive { // indexed-tree layers' documented under-count (see // `estimated_sum_trees_for_value_tree_type`). let item_value = if estimated_costs_only_with_layer_info.is_some() { - vec![0u8; INDEX_ONLY_ITEM_ESTIMATED_VALUE_SIZE as usize] + vec![0u8; estimated_item_value_size as usize] } else { let (document, _) = document_and_contract_info .owned_document_info @@ -523,12 +532,19 @@ impl Drive { .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( "indexOnly terminal insert needs a document outside estimation mode", )))?; - crate::drive::document::index_only_row_commitment( + // The commitment first, then the type's entry payload (empty + // without an `entryPayload` declaration). + let mut item_value = crate::drive::document::index_only_row_commitment( document, document_and_contract_info.document_type, platform_version, )? - .to_vec() + .to_vec(); + item_value.extend(encode_index_only_entry_payload( + document, + document_and_contract_info.document_type, + )?); + item_value }; let key_element_info = match &member_key { @@ -561,7 +577,7 @@ impl Drive { .document_type .unique_id_for_storage() .to_vec(), - max_size: DEFAULT_HASH_SIZE_U8, + max_size: member_key_max_size, }, item_space, )), diff --git a/packages/rs-drive/src/drive/document/mod.rs b/packages/rs-drive/src/drive/document/mod.rs index aa574966e1f..6ad61c2253a 100644 --- a/packages/rs-drive/src/drive/document/mod.rs +++ b/packages/rs-drive/src/drive/document/mod.rs @@ -78,6 +78,17 @@ pub mod index_only; #[cfg(any(feature = "server", feature = "verify"))] pub mod index_only_row_commitment; +/// The entry payload of an indexOnly document type: the value slot after +/// the row commitment, its encoding and its fee-estimation bounds. +#[cfg(any(feature = "server", feature = "verify"))] +pub mod index_only_entry_payload; + +#[cfg(any(feature = "server", feature = "verify"))] +pub use index_only_entry_payload::{ + decode_index_only_entry_payload, encode_index_only_entry_payload, + encode_index_only_entry_payload_value, index_only_entry_payload_max_size, + index_only_item_estimated_value_size, +}; #[cfg(any(feature = "server", feature = "verify"))] pub use index_only_row_commitment::index_only_row_commitment; #[cfg(feature = "server")] diff --git a/packages/rs-drive/src/drive/document/ranked_index_tree_type.rs b/packages/rs-drive/src/drive/document/ranked_index_tree_type.rs index 6b09862ca28..a514cc63581 100644 --- a/packages/rs-drive/src/drive/document/ranked_index_tree_type.rs +++ b/packages/rs-drive/src/drive/document/ranked_index_tree_type.rs @@ -230,6 +230,7 @@ mod tests { ranked_averageable, terminal: None, preallocated: false, + flat: false, } } diff --git a/packages/rs-drive/src/query/chained_document_query/mod.rs b/packages/rs-drive/src/query/chained_document_query/mod.rs index 21d8105c249..42080512394 100644 --- a/packages/rs-drive/src/query/chained_document_query/mod.rs +++ b/packages/rs-drive/src/query/chained_document_query/mod.rs @@ -265,7 +265,7 @@ impl<'a> DriveDocumentQuery<'a> { // The resolved index must carry the join property, so every // synthesized inner projection provably carries its value. let index = self.index_only_query_index(platform_version)?; - let index_carries_join_property = index.terminal.as_deref() == Some(join_property) + let index_carries_join_property = index.terminal_contains(join_property) || index .properties .iter() diff --git a/packages/rs-drive/src/query/composite_document_query/mod.rs b/packages/rs-drive/src/query/composite_document_query/mod.rs index ce26e8c4c25..c3ee3fabdff 100644 --- a/packages/rs-drive/src/query/composite_document_query/mod.rs +++ b/packages/rs-drive/src/query/composite_document_query/mod.rs @@ -513,7 +513,7 @@ impl<'a> DriveDocumentQuery<'a> { // carries, so the property must sit on that index. if source_is_index_only_query { let carries = |index: &dpp::data_contract::document_type::Index| { - index.terminal.as_deref() == Some(binding.source_property.as_str()) + index.terminal_contains(&binding.source_property) || index .properties .iter() @@ -656,7 +656,7 @@ impl<'a> DriveDocumentQuery<'a> { // The lookup's own field must be provable positionally: // the resolved index has to carry it. let index = shape.index_only_query_index(platform_version)?; - let carried = index.terminal.as_deref() == Some(binding.field.as_str()) + let carried = index.terminal_contains(&binding.field) || index .properties .iter() @@ -744,7 +744,9 @@ impl<'a> DriveDocumentQuery<'a> { .collect(); if sub_query.document_type.index_only() { let index = shape.index_only_query_index(platform_version)?; - let terminal_is_bound = index.terminal.as_deref() == Some(binding.field.as_str()); + // A composite terminal is bound only through every component; + // a single-component terminal through its one field. + let terminal_is_bound = index.single_terminal() == Some(binding.field.as_str()); let prefix_fixed = index .properties .iter() @@ -1381,13 +1383,14 @@ impl<'a> DriveDocumentQuery<'a> { let index = query.index_only_query_index(platform_version)?; return trios .into_iter() - .map(|(path, key, _)| { + .map(|(path, key, element)| { synthesize_index_only_document( query.contract.id(), query.document_type, index, &path, &key, + Some(&element), ) }) .collect(); diff --git a/packages/rs-drive/src/query/index_only_synthesis.rs b/packages/rs-drive/src/query/index_only_synthesis.rs index 2ad462c278b..569792b9f1a 100644 --- a/packages/rs-drive/src/query/index_only_synthesis.rs +++ b/packages/rs-drive/src/query/index_only_synthesis.rs @@ -33,13 +33,14 @@ //! Fail-closed: any arity or property-name mismatch between the trio and //! the index the query resolved is an error, never a partial document. +use crate::drive::document::{decode_index_only_entry_payload, INDEX_ONLY_ROW_COMMITMENT_SIZE}; use crate::error::drive::DriveError; use crate::error::Error; use crate::query::{index_admissible_for_skip_if_absent, DriveDocumentQuery}; use crate::verify::RootHash; use dpp::data_contract::accessors::v0::DataContractV0Getters; -use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; -use dpp::data_contract::document_type::{DocumentTypeRef, Index}; +use dpp::data_contract::document_type::accessors::{DocumentTypeV0Getters, DocumentTypeV2Getters}; +use dpp::data_contract::document_type::{DocumentPropertyType, DocumentTypeRef, Index}; use dpp::document::{Document, DocumentV0}; use dpp::identifier::Identifier; use dpp::platform_value::btreemap_extensions::BTreeValueMapInsertionPathHelper; @@ -63,6 +64,13 @@ pub(crate) struct IndexOnlyTerminalRoute<'a> { /// `(position, clause)` of a range / `in` clause on a prefix /// property. When present the terminal clause is always an equality. pub prefix_pivot: Option<(usize, &'a crate::query::WhereClause)>, + /// COMPOSITE terminals only: the equality clauses on the terminal's + /// leading components, in component order (`terminal_clause` is `None` + /// on a composite terminal). + pub terminal_equalities: Vec<&'a crate::query::WhereClause>, + /// COMPOSITE terminals only: the one range / `in` clause on the first + /// component after the equality-bound ones, if any. + pub terminal_tail: Option<&'a crate::query::WhereClause>, } impl DriveDocumentQuery<'_> { @@ -195,27 +203,13 @@ impl DriveDocumentQuery<'_> { return Ok(None); } - let terminal = - index - .terminal - .as_deref() - .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( - "a terminal-using match implies an indexOnly index", - )))?; - let terminal_clause = self - .internal_clauses - .equal_clauses - .get(terminal) - .or(match &self.internal_clauses.range_clause { - Some(range_clause) if range_clause.field == terminal => Some(range_clause), - _ => None, - }) - .or_else(|| { - self.internal_clauses - .in_clauses - .iter() - .find(|in_clause| in_clause.field == terminal) - }); + let components = index.terminal_components(); + if components.is_empty() { + return Err(Error::Drive(DriveError::CorruptedCodeExecution( + "a terminal-using match implies an indexOnly index", + ))); + } + let is_component = |field: &str| components.iter().any(|component| component == field); let shape_error = |message: &str| { Error::Query(crate::error::query::QuerySyntaxError::Unsupported( @@ -223,6 +217,110 @@ impl DriveDocumentQuery<'_> { )) }; + // A single-component terminal carries at most one clause, on that + // component. A composite terminal binds its components in order: + // equality clauses on the leading components, then at most one + // range or `in` clause on the next one, nothing on the rest — the + // shape that lowers onto one contiguous range of member keys. + let (terminal_clause, terminal_equalities, terminal_tail) = match index.single_terminal() { + Some(terminal) => { + let terminal_clause = self + .internal_clauses + .equal_clauses + .get(terminal) + .or(match &self.internal_clauses.range_clause { + Some(range_clause) if range_clause.field == terminal => Some(range_clause), + _ => None, + }) + .or_else(|| { + self.internal_clauses + .in_clauses + .iter() + .find(|in_clause| in_clause.field == terminal) + }); + (terminal_clause, Vec::new(), None) + } + None => { + let mut equalities: Vec<&crate::query::WhereClause> = Vec::new(); + for component in components { + match self.internal_clauses.equal_clauses.get(component.as_str()) { + Some(clause) => equalities.push(clause), + None => break, + } + } + let bound = equalities.len(); + if components.iter().skip(bound).any(|component| { + self.internal_clauses + .equal_clauses + .contains_key(component.as_str()) + }) { + return Err(shape_error( + "equality clauses on a composite indexOnly terminal must bind its \ + components contiguously from the first one: a component cannot be \ + bound while an earlier one is not", + )); + } + // All terminal components share one lexicographically ordered + // member key. After ignoring equality-bound fields, ORDER BY + // must follow the remaining components without gaps, in one + // direction; reversing the key reverses every component. + let mut direction = None; + for (position, order) in self + .order_by + .values() + .filter(|order| { + is_component(&order.field) + && !self + .internal_clauses + .equal_clauses + .contains_key(&order.field) + }) + .enumerate() + { + if components.get(bound + position) != Some(&order.field) { + return Err(shape_error( + "orderBy on a composite indexOnly terminal must follow its \ + unbound components contiguously from the first one", + )); + } + if direction.is_some_and(|ascending| ascending != order.ascending) { + return Err(shape_error( + "orderBy on unbound composite indexOnly terminal components \ + must use the same direction", + )); + } + direction = Some(order.ascending); + } + let tail_candidates: Vec<&crate::query::WhereClause> = self + .internal_clauses + .range_clause + .iter() + .chain(self.internal_clauses.in_clauses.iter()) + .filter(|clause| is_component(&clause.field)) + .collect(); + let tail = match tail_candidates.as_slice() { + [] => None, + [clause] => { + if components.get(bound).map(String::as_str) != Some(clause.field.as_str()) + { + return Err(shape_error( + "a range or `in` clause on a composite indexOnly terminal must \ + sit on the first component after the equality-bound ones", + )); + } + Some(*clause) + } + _ => { + return Err(shape_error( + "a composite indexOnly terminal supports at most one range or `in` \ + clause, on the first component after the equality-bound ones", + )) + } + }; + (None, equalities, tail) + } + }; + // The one non-equality, non-terminal clause — the prefix pivot // candidate. (Field coverage is the matcher's job; PLACEMENT of // non-equality clauses is the shape rule here.) @@ -239,7 +337,7 @@ impl DriveDocumentQuery<'_> { .iter() .chain(self.internal_clauses.in_clauses.iter()) { - if clause.field == terminal { + if is_component(&clause.field) { continue; } let Some(position) = position_of(&clause.field) else { @@ -275,9 +373,11 @@ impl DriveDocumentQuery<'_> { orderBy limited to the index", )); } - if let Some(terminal_clause) = terminal_clause { - if terminal_clause.operator.is_range() && !self.order_by.contains_key(terminal) - { + let ranged: Option<&crate::query::WhereClause> = terminal_clause + .filter(|clause| clause.operator.is_range()) + .or(terminal_tail.filter(|clause| clause.operator.is_range())); + if let Some(ranged) = ranged { + if !self.order_by.contains_key(ranged.field.as_str()) { return Err(Error::Query( crate::error::query::QuerySyntaxError::MissingOrderByForRange( "a range or `in` clause on an indexOnly terminal property \ @@ -291,8 +391,10 @@ impl DriveDocumentQuery<'_> { // Mixed shape: everything above the pivot equality-bound, // everything below it unconstrained, terminal clause an // equality, pivot ordered by. - let terminal_is_equality = - self.internal_clauses.equal_clauses.contains_key(terminal); + let terminal_is_equality = match index.single_terminal() { + Some(terminal) => self.internal_clauses.equal_clauses.contains_key(terminal), + None => terminal_equalities.len() == components.len(), + }; if !terminal_is_equality { return Err(shape_error( "a range or `in` clause on an indexOnly prefix property requires \ @@ -334,6 +436,8 @@ impl DriveDocumentQuery<'_> { index, terminal_clause, prefix_pivot, + terminal_equalities, + terminal_tail, })) } @@ -359,6 +463,8 @@ impl DriveDocumentQuery<'_> { index, terminal_clause, prefix_pivot, + terminal_equalities, + terminal_tail, } = route; let direction_for = |field: &str, fallback: bool| { @@ -367,8 +473,8 @@ impl DriveDocumentQuery<'_> { .map(|order_clause| order_clause.ascending) .unwrap_or(fallback) }; - let terminal_query = match terminal_clause { - Some(terminal_clause) => { + let terminal_query = match (index.single_terminal(), terminal_clause) { + (Some(_), Some(terminal_clause)) => { let left_to_right = if terminal_clause.operator.is_range() { direction_for(terminal_clause.field.as_str(), true) } else { @@ -383,16 +489,19 @@ impl DriveDocumentQuery<'_> { } // First keyset page: no cursor clause yet — every member key // in the terminal's orderBy direction. - None => { - let terminal = index.terminal.as_deref().ok_or(Error::Drive( - DriveError::CorruptedCodeExecution( - "terminal-route selection guarantees an indexOnly index", - ), - ))?; + (Some(terminal), None) => { let mut query = grovedb::Query::new_with_direction(direction_for(terminal, true)); query.insert_all(); query } + // Composite terminal: the bound components form a key prefix, + // the tail clause (or its absence) a range under it. + (None, _) => self.composite_member_key_query( + index, + terminal_equalities, + *terminal_tail, + platform_version, + )?, }; let mut path = document_type_path; @@ -468,7 +577,13 @@ impl DriveDocumentQuery<'_> { )?); } match prefix_pivot { - None => path.push(vec![0]), + None => { + // A flat index keeps its entries under its own level. + if let Some(flat_key) = index.flat_level_key() { + path.push(flat_key.into_bytes()); + } + path.push(vec![0]); + } Some((pivot_position, _)) => { path.push(index.properties[*pivot_position].name.as_bytes().to_vec()) } @@ -480,6 +595,186 @@ impl DriveDocumentQuery<'_> { )) } + /// The member-key query of a composite terminal: the equality-bound + /// leading components serialize to a key prefix, and the tail clause + /// (a range or `in` on the next component), or its absence, becomes a + /// contiguous key range under that prefix. Every component but the + /// last is fixed width (the parser enforces it), so a bound on a + /// non-last component covers every key that continues past it: the + /// upper bound of "all keys under P" is P padded with 0xFF to the + /// 255-byte key cap, which every key with prefix P sorts at or below. + /// A bound on the LAST component addresses the key itself. + fn composite_member_key_query( + &self, + index: &Index, + terminal_equalities: &[&crate::query::WhereClause], + terminal_tail: Option<&crate::query::WhereClause>, + platform_version: &PlatformVersion, + ) -> Result { + use crate::query::WhereOperator; + use dpp::data_contract::document_type::methods::DocumentTypeV0Methods; + + const MAX_KEY_LENGTH: usize = u8::MAX as usize; + let components = index.terminal_components(); + let bound = terminal_equalities.len(); + let mut prefix: Vec = Vec::new(); + for clause in terminal_equalities { + prefix.extend(self.document_type.serialize_value_for_key( + &clause.field, + &clause.value, + platform_version, + )?); + } + let direction_field = terminal_tail + .map(|clause| clause.field.as_str()) + .or_else(|| components.get(bound).map(String::as_str)); + let left_to_right = direction_field + .and_then(|field| self.order_by.get(field)) + .map(|order_clause| order_clause.ascending) + .unwrap_or(true); + let mut query = grovedb::Query::new_with_direction(left_to_right); + + if bound == components.len() { + query.insert_key(prefix); + return Ok(query); + } + + let pad_max = |mut key: Vec| { + key.resize(key.len().max(MAX_KEY_LENGTH), 0xFF); + key + }; + let Some(tail_component) = components.get(bound) else { + return Err(Error::Drive(DriveError::CorruptedCodeExecution( + "a composite terminal with unbound components has a next component", + ))); + }; + let tail_is_last = bound + 1 == components.len(); + let encode = |value: &Value| -> Result, Error> { + let mut key = prefix.clone(); + key.extend(self.document_type.serialize_value_for_key( + tail_component, + value, + platform_version, + )?); + Ok(key) + }; + // Inclusive upper bound of every key whose tail component equals + // `value`: the key itself on the last component, else the padded + // continuation. + let upper_inclusive = |value: &Value| -> Result, Error> { + let key = encode(value)?; + Ok(if tail_is_last { key } else { pad_max(key) }) + }; + // Exclusive lower bound of every key whose tail component exceeds + // `value`, for the range-after variants. + let lower_exclusive = upper_inclusive; + + let Some(tail) = terminal_tail else { + if prefix.is_empty() { + query.insert_all(); + } else { + query.insert_range_inclusive(prefix.clone()..=pad_max(prefix)); + } + return Ok(query); + }; + let between_bounds = |value: &Value| -> Result<(Value, Value), Error> { + match value { + Value::Array(values) if values.len() == 2 => { + Ok((values[0].clone(), values[1].clone())) + } + _ => Err(Error::Query( + crate::error::query::QuerySyntaxError::InvalidBetweenClause( + "when using between operator you must provide a tuple array of values", + ), + )), + } + }; + match tail.operator { + WhereOperator::Equal => { + // Unreachable through selection (equalities are consumed + // above); lowered as the key prefix anyway. + let key = encode(&tail.value)?; + query.insert_range_inclusive(key.clone()..=pad_max(key)); + } + WhereOperator::GreaterThan => { + query.insert_range_after_to_inclusive( + lower_exclusive(&tail.value)?..=pad_max(prefix.clone()), + ); + } + WhereOperator::GreaterThanOrEquals => { + query.insert_range_inclusive(encode(&tail.value)?..=pad_max(prefix.clone())); + } + WhereOperator::LessThan => { + if prefix.is_empty() { + query.insert_range_to(..encode(&tail.value)?); + } else { + query.insert_range(prefix.clone()..encode(&tail.value)?); + } + } + WhereOperator::LessThanOrEquals => { + if prefix.is_empty() { + query.insert_range_to_inclusive(..=upper_inclusive(&tail.value)?); + } else { + query.insert_range_inclusive(prefix.clone()..=upper_inclusive(&tail.value)?); + } + } + WhereOperator::Between => { + let (low, high) = between_bounds(&tail.value)?; + query.insert_range_inclusive(encode(&low)?..=upper_inclusive(&high)?); + } + WhereOperator::BetweenExcludeBounds => { + let (low, high) = between_bounds(&tail.value)?; + query.insert_range_after_to(lower_exclusive(&low)?..encode(&high)?); + } + WhereOperator::BetweenExcludeLeft => { + let (low, high) = between_bounds(&tail.value)?; + query.insert_range_after_to_inclusive( + lower_exclusive(&low)?..=upper_inclusive(&high)?, + ); + } + WhereOperator::BetweenExcludeRight => { + let (low, high) = between_bounds(&tail.value)?; + query.insert_range(encode(&low)?..encode(&high)?); + } + WhereOperator::In => { + let in_values = tail.in_values().into_data_with_error()??; + for value in in_values.iter() { + let key = encode(value)?; + if tail_is_last { + query.insert_key(key); + } else { + query.insert_range_inclusive(key.clone()..=pad_max(key)); + } + } + } + WhereOperator::StartsWith => { + return Err(Error::Query( + crate::error::query::QuerySyntaxError::Unsupported( + "startsWith is not supported on a composite indexOnly terminal \ + component" + .to_string(), + ), + )); + } + } + Ok(query) + } + + /// Whether a query with no clauses at all is a scan of a FLAT index + /// rather than a by-id query: an indexOnly type with a flat index has + /// somewhere for "everything" to land (its flat level, served by the + /// generic match in `index_only_route`), while a type without one has + /// no primary-key tree to walk. + pub(crate) fn index_only_flat_scan_applies(&self) -> bool { + self.internal_clauses.is_empty() + && self.start_at.is_none() + && self + .document_type + .indexes() + .values() + .any(|index| index.is_flat()) + } + /// The index an indexOnly query resolves to — the generic matcher /// when it can serve the query, else the terminal-clause route. Used /// by synthesis (which must decode trios against the same index the @@ -545,6 +840,24 @@ impl DriveDocumentQuery<'_> { // the prover and the no-proof executor fail before // building a path query the synthesis side would refuse. Self::refuse_bucketed_index_only_synthesis(index)?; + // A generic match on a FLAT index binds nothing (the + // index has no properties to bind), so it is a scan of + // every member under the flat level. + if let Some(flat_key) = index.flat_level_key() { + let mut path = document_type_path.to_vec(); + path.push(flat_key.into_bytes()); + path.push(vec![0]); + // An orderBy naming a component makes the matcher + // report the terminal as used, which sends the query + // down the terminal route instead; the generic match + // only ever sees the unordered scan. + let mut query = grovedb::Query::new_with_direction(true); + query.insert_all(); + return Ok(Some(grovedb::PathQuery::new( + path, + grovedb::SizedQuery::new(query, self.limit, self.offset), + ))); + } Ok(None) } crate::query::BestIndexOutcome::NoIndexMatches(no_index_error) => { @@ -588,14 +901,15 @@ impl DriveDocumentQuery<'_> { let index = self.index_only_query_index(platform_version)?; let documents = proved_key_values .into_iter() - .filter_map(|(path, key, element)| element.map(|_| (path, key))) - .map(|(path, key)| { + .filter_map(|(path, key, element)| element.map(|element| (path, key, element))) + .map(|(path, key, element)| { synthesize_index_only_document( self.contract.id(), self.document_type, index, &path, &key, + Some(&element), ) }) .collect::, Error>>()?; @@ -663,13 +977,14 @@ impl DriveDocumentQuery<'_> { let documents = elements .to_path_key_elements() .into_iter() - .map(|(path, key, _element)| { + .map(|(path, key, element)| { synthesize_index_only_document( self.contract.id(), self.document_type, index, &path, &key, + Some(&element), ) }) .collect::, Error>>()?; @@ -692,9 +1007,9 @@ pub fn index_only_proof_index<'a>(document_type: &'a DocumentTypeRef) -> Result< .indexes() .values() .find(|index| { - let carries_owner = index.terminal.as_deref() == Some(OWNER_ID) + let carries_owner = index.terminal_contains(OWNER_ID) || index.properties.iter().any(|p| p.name == OWNER_ID); - let carries_created_at = index.terminal.as_deref() == Some(CREATED_AT) + let carries_created_at = index.terminal_contains(CREATED_AT) || index.properties.iter().any(|p| p.name == CREATED_AT); carries_owner && !carries_created_at && !index.skip_if_absent }) @@ -766,16 +1081,23 @@ pub fn index_only_entry_path_and_key_from_values( path.push(property.name.as_bytes().to_vec()); path.push(encoded_value_for(&property.name)?); } + // A flat index keeps its entries under its own level, between the + // doctype and the `0` bucket. + if let Some(flat_key) = index.flat_level_key() { + path.push(flat_key.as_bytes().to_vec()); + } path.push(vec![0]); - let terminal = - index - .terminal - .as_deref() - .ok_or(Error::Drive(DriveError::CorruptedCodeExecution( - "index_only_entry_path_and_key_from_values requires an indexOnly index", - )))?; - let member_key = encoded_value_for(terminal)?; + if index.terminal.is_none() { + return Err(Error::Drive(DriveError::CorruptedCodeExecution( + "index_only_entry_path_and_key_from_values requires an indexOnly index", + ))); + } + // The member key: the terminal components' encoded values, concatenated. + let mut member_key = Vec::new(); + for component in index.terminal_components() { + member_key.extend(encoded_value_for(component)?); + } Ok((path, member_key)) } @@ -809,21 +1131,30 @@ pub fn index_only_transition_entry_path_query( /// /// `path` is the grove path of the entry's parent (ending with the `0` /// storage marker); `member_key` is the entry's key (the terminal -/// property's encoded value). +/// components' encoded values, concatenated); `element` is the proved +/// entry item, required on a type with an `entryPayload` (the payload is +/// decoded off it) and ignored otherwise. pub fn synthesize_index_only_document( contract_id: Identifier, document_type: DocumentTypeRef, index: &Index, path: &[Vec], member_key: &[u8], + element: Option<&grovedb::Element>, ) -> Result { use dpp::document::property_names::{CREATED_AT, OWNER_ID}; let corrupted = |message: &'static str| Error::Drive(DriveError::CorruptedCodeExecution(message)); - // The path must end [, , …, , , [0]]. - let expected_suffix_len = index.properties.len() * 2 + 1; + // The path must end [, , …, , , [0]] — or, + // for a flat index, [, [0]]. + let flat_key = index.flat_level_key(); + let expected_suffix_len = if flat_key.is_some() { + 2 + } else { + index.properties.len() * 2 + 1 + }; if path.len() < expected_suffix_len { return Err(corrupted( "indexOnly synthesis: proved path is shorter than the resolved index's shape", @@ -835,6 +1166,14 @@ pub fn synthesize_index_only_document( "indexOnly synthesis: proved path does not end at the 0 storage marker", )); } + if let Some(flat_key) = &flat_key { + if suffix[0].as_slice() != flat_key.as_bytes() { + return Err(corrupted( + "indexOnly synthesis: proved path does not sit under the resolved flat \ + index's level — refusing to mislabel a value", + )); + } + } let mut properties: BTreeMap = BTreeMap::new(); let mut owner_id: Option = None; @@ -849,14 +1188,9 @@ pub fn synthesize_index_only_document( ); } CREATED_AT => { - created_at = Some( - dpp::data_contract::document_type::DocumentPropertyType::decode_date_timestamp( - encoded, - ) - .ok_or(corrupted( - "indexOnly synthesis: $createdAt key bytes are not a timestamp", - ))?, - ); + created_at = Some(DocumentPropertyType::decode_date_timestamp(encoded).ok_or( + corrupted("indexOnly synthesis: $createdAt key bytes are not a timestamp"), + )?); } name => { let property = document_type @@ -865,10 +1199,20 @@ pub fn synthesize_index_only_document( .ok_or(corrupted( "indexOnly synthesis: index names a property the document type lacks", ))?; - let value = property - .property_type - .decode_value_for_tree_keys(encoded) - .map_err(|e| Error::Protocol(Box::new(e)))?; + // Every indexed property of an indexOnly type is required, + // so an empty key never encodes an absent value: for a byte + // array it is the empty array itself, which the tree-key + // decoder would otherwise read back as the null sentinel. + let value = if encoded.is_empty() + && matches!(property.property_type, DocumentPropertyType::ByteArray(_)) + { + Value::Bytes(Vec::new()) + } else { + property + .property_type + .decode_value_for_tree_keys(encoded) + .map_err(|e| Error::Protocol(Box::new(e)))? + }; // A flattened name like `profile.targetId` must come back // as a nested `profile` map, not as a dotted top-level key // — field access, schema serialization and index encoding @@ -892,11 +1236,78 @@ pub fn synthesize_index_only_document( assign(&index_property.name, &suffix[position * 2 + 1])?; } - let terminal = index.terminal.as_deref().ok_or(corrupted( - "indexOnly synthesis requires an indexOnly index (terminal is always Some \ - after parse normalization)", - ))?; - assign(terminal, member_key)?; + // The member key splits back into the terminal's components: every + // component but the last is fixed width (the parser enforces it), and + // the last takes the remainder. + let components = index.terminal_components(); + if components.is_empty() { + return Err(corrupted( + "indexOnly synthesis requires an indexOnly index (terminal is always Some \ + after parse normalization)", + )); + } + let mut terminal_parts: Vec<(&str, &[u8])> = Vec::with_capacity(components.len()); + let mut cursor = 0usize; + for (position, component) in components.iter().enumerate() { + let is_last = position + 1 == components.len(); + let bytes: &[u8] = + if is_last { + member_key.get(cursor..).ok_or(corrupted( + "indexOnly synthesis: member key is shorter than its leading components", + ))? + } else { + let width = + if component == OWNER_ID { + 32usize + } else { + let property = + document_type + .flattened_properties() + .get(component) + .ok_or(corrupted( + "indexOnly synthesis: terminal names a property the document type lacks", + ))?; + usize::from(property.property_type.fixed_tree_key_width().ok_or(corrupted( + "indexOnly synthesis: a leading terminal component must be fixed width", + ))?) + }; + let slice = member_key.get(cursor..cursor + width).ok_or(corrupted( + "indexOnly synthesis: member key is shorter than its leading components", + ))?; + cursor += width; + slice + }; + assign(component, bytes)?; + terminal_parts.push((component.as_str(), bytes)); + } + + // The entry payload — the type's value slot — rides in the item after + // the row commitment. + if !document_type.entry_payload().is_empty() { + let Some(element) = element else { + return Err(corrupted( + "indexOnly synthesis: a type with an entryPayload needs the proved element", + )); + }; + let item = match element { + grovedb::Element::Item(bytes, _) | grovedb::Element::ItemWithSumItem(bytes, _, _) => { + bytes + } + _ => { + return Err(corrupted( + "indexOnly synthesis: the proved entry is not an item element", + )) + } + }; + let payload = item + .get(INDEX_ONLY_ROW_COMMITMENT_SIZE as usize..) + .ok_or(corrupted( + "indexOnly synthesis: the proved entry item is shorter than the row commitment", + ))?; + for (name, value) in decode_index_only_entry_payload(document_type, payload)? { + properties.insert(name, value); + } + } let owner_id = owner_id.ok_or(Error::Query( crate::error::query::QuerySyntaxError::Unsupported( @@ -929,9 +1340,11 @@ pub fn synthesize_index_only_document( frame(&mut id_preimage, &suffix[position * 2 + 1]); } } - if terminal != OWNER_ID { - frame(&mut id_preimage, terminal.as_bytes()); - frame(&mut id_preimage, member_key); + for (component, bytes) in terminal_parts { + if component != OWNER_ID { + frame(&mut id_preimage, component.as_bytes()); + frame(&mut id_preimage, bytes); + } } let id = Identifier::new(hash_double(id_preimage)); diff --git a/packages/rs-drive/src/query/mod.rs b/packages/rs-drive/src/query/mod.rs index 659d768668f..1a5175a2f21 100644 --- a/packages/rs-drive/src/query/mod.rs +++ b/packages/rs-drive/src/query/mod.rs @@ -435,7 +435,7 @@ impl InternalClauses { { roles.index_property = true; } - if index.terminal.as_deref() == Some(field) { + if index.terminal_contains(field) { roles.terminal = true; } if roles.index_property && roles.terminal { @@ -1936,7 +1936,10 @@ impl<'a> DriveDocumentQuery<'a> { // addressed by document id, so a by-id query has no tree to land on. { use dpp::data_contract::document_type::accessors::DocumentTypeV2Getters; - if self.document_type.index_only() && self.is_for_primary_key() { + if self.document_type.index_only() + && self.is_for_primary_key() + && !self.index_only_flat_scan_applies() + { return Err(Error::Query(QuerySyntaxError::Unsupported( "indexOnly documents cannot be fetched by id: there is no primary-key \ tree; query through one of the type's indexes" @@ -2153,7 +2156,10 @@ impl<'a> DriveDocumentQuery<'a> { // addressed by document id, so a by-id query has no tree to land on. { use dpp::data_contract::document_type::accessors::DocumentTypeV2Getters; - if self.document_type.index_only() && self.is_for_primary_key() { + if self.document_type.index_only() + && self.is_for_primary_key() + && !self.index_only_flat_scan_applies() + { return Err(Error::Query(QuerySyntaxError::Unsupported( "indexOnly documents cannot be fetched by id: there is no primary-key \ tree; query through one of the type's indexes" @@ -2961,7 +2967,12 @@ impl<'a> DriveDocumentQuery<'a> { // the route (no primary-key tree; keyset pagination) — let // them reach it instead of preempting with the coverage // refusal below, which would misdescribe the problem. - if !self.is_for_primary_key() && self.start_at.is_none() { + // A clause-free flat scan is classified as a primary-key + // query, but still synthesizes an index projection and must + // pass the same coverage check as a filtered query. + if (!self.is_for_primary_key() || self.index_only_flat_scan_applies()) + && self.start_at.is_none() + { let index = self.index_only_query_index(platform_version)?; let covers_every_property = self .document_type @@ -2971,7 +2982,8 @@ impl<'a> DriveDocumentQuery<'a> { !matches!(property.property_type, DocumentPropertyType::Object(_)) }) .all(|(name, _)| { - index.terminal.as_deref() == Some(name.as_str()) + index.terminal_contains(name) + || self.document_type.entry_payload().contains(name.as_str()) || index .properties .iter() diff --git a/packages/rs-drive/src/verify/chained_document/verify_chained_documents_proof/v0/mod.rs b/packages/rs-drive/src/verify/chained_document/verify_chained_documents_proof/v0/mod.rs index 6777d32b7e0..daaa20e1750 100644 --- a/packages/rs-drive/src/verify/chained_document/verify_chained_documents_proof/v0/mod.rs +++ b/packages/rs-drive/src/verify/chained_document/verify_chained_documents_proof/v0/mod.rs @@ -39,13 +39,14 @@ impl DriveDocumentQuery<'_> { let bootstrap_documents = bootstrap_trios .into_iter() .filter(|(_, _, element)| element.is_some()) - .map(|(path, key, _)| { + .map(|(path, key, element)| { synthesize_index_only_document( self.contract.id(), self.document_type, index, &path, &key, + element.as_ref(), ) }) .collect::, Error>>()?; @@ -92,6 +93,7 @@ impl DriveDocumentQuery<'_> { index, &path, &key, + Some(&element), )?); } Some(segment) if segment == outer_type_name => { diff --git a/packages/rs-drive/src/verify/state_transition/verify_state_transition_was_executed_with_proof/v0/mod.rs b/packages/rs-drive/src/verify/state_transition/verify_state_transition_was_executed_with_proof/v0/mod.rs index f67243a89ba..4b2151f00f7 100644 --- a/packages/rs-drive/src/verify/state_transition/verify_state_transition_was_executed_with_proof/v0/mod.rs +++ b/packages/rs-drive/src/verify/state_transition/verify_state_transition_was_executed_with_proof/v0/mod.rs @@ -290,7 +290,13 @@ impl Drive { document_type, platform_version, )?; - if payload != expected_commitment { + // The commitment is the item's + // first 32 bytes; a type's entry + // payload follows it and is covered + // by the commitment. + if payload.get(..expected_commitment.len()) + != Some(expected_commitment.as_slice()) + { return Err(Error::Proof(ProofError::IncorrectProof(format!( "the proved indexOnly entry's row commitment does not match the created document {}: the entry belongs to a different row", create_transition.base().id() diff --git a/packages/rs-drive/tests/supporting_files/contract/index-only-scalar-terminal/index-only-scalar-terminal-contract.json b/packages/rs-drive/tests/supporting_files/contract/index-only-scalar-terminal/index-only-scalar-terminal-contract.json new file mode 100644 index 00000000000..03ef1373fe9 --- /dev/null +++ b/packages/rs-drive/tests/supporting_files/contract/index-only-scalar-terminal/index-only-scalar-terminal-contract.json @@ -0,0 +1,339 @@ +{ + "$formatVersion": "0", + "id": "CD6KsgzPhL8xPqHjvg4XWfDRNoGRURiPknAMiApkjgHR", + "ownerId": "ASaUNbfe4QBnWU3izvnZHagrxM3Rwhjz5hBmjWZg8RSX", + "version": 1, + "documentSchemas": { + "answer": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byRequest", + "properties": [ + { + "requestId": "asc" + }, + { + "$ownerId": "asc" + } + ], + "terminal": "payload" + } + ], + "properties": { + "requestId": { + "type": "array", + "byteArray": true, + "minItems": 20, + "maxItems": 20, + "position": 0 + }, + "payload": { + "type": "array", + "byteArray": true, + "minItems": 33, + "maxItems": 33, + "position": 1 + } + }, + "required": [ + "requestId", + "payload" + ], + "additionalProperties": false + }, + "vote": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byPoll", + "properties": [ + { + "pollId": "asc" + }, + { + "$ownerId": "asc" + } + ], + "terminal": "choice" + }, + { + "name": "byChoice", + "properties": [ + { + "pollId": "asc" + }, + { + "choice": "asc" + } + ], + "terminal": "$ownerId", + "countable": true, + "rangeCountable": true, + "rankedCountable": true + } + ], + "properties": { + "pollId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "position": 0 + }, + "choice": { + "type": "string", + "maxLength": 16, + "position": 1 + } + }, + "required": [ + "pollId", + "choice" + ], + "additionalProperties": false + }, + "rating": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byPost", + "properties": [ + { + "postId": "asc" + }, + { + "$ownerId": "asc" + } + ], + "terminal": "stars" + } + ], + "properties": { + "postId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "position": 0 + }, + "stars": { + "type": "integer", + "minimum": 1, + "maximum": 5, + "position": 1 + } + }, + "required": [ + "postId", + "stars" + ], + "additionalProperties": false + }, + "loginKeyResponse": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byRequest", + "terminal": [ + "appEphemeralPubKeyHash", + "$ownerId" + ] + } + ], + "entryPayload": [ + "walletEphemeralPubKey", + "encryptedPayload" + ], + "properties": { + "appEphemeralPubKeyHash": { + "type": "array", + "byteArray": true, + "minItems": 20, + "maxItems": 20, + "position": 0 + }, + "walletEphemeralPubKey": { + "type": "array", + "byteArray": true, + "minItems": 33, + "maxItems": 33, + "position": 1 + }, + "encryptedPayload": { + "type": "array", + "byteArray": true, + "minItems": 60, + "maxItems": 572, + "position": 2 + } + }, + "required": [ + "appEphemeralPubKeyHash", + "walletEphemeralPubKey", + "encryptedPayload" + ], + "additionalProperties": false + }, + "payloadValues": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byOwner", + "terminal": "$ownerId" + } + ], + "entryPayload": [ + "bytes", + "text" + ], + "properties": { + "bytes": { + "type": "array", + "byteArray": true, + "minItems": 0, + "maxItems": 16, + "position": 0 + }, + "text": { + "type": "string", + "maxLength": 16, + "position": 1 + } + }, + "required": [ + "bytes", + "text" + ], + "additionalProperties": false + }, + "tagged": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byOwner", + "terminal": "$ownerId" + }, + { + "name": "byTag", + "properties": [ + { + "tag": "asc" + } + ], + "skipIfAbsent": true + } + ], + "properties": { + "tag": { + "type": "string", + "maxLength": 16, + "position": 0 + } + }, + "additionalProperties": false + }, + "reaction": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byPostKind", + "properties": [ + { + "postId": "asc" + } + ], + "terminal": [ + "kind", + "$ownerId" + ] + } + ], + "properties": { + "postId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "position": 0 + }, + "kind": { + "type": "integer", + "minimum": 0, + "maximum": 10, + "position": 1 + } + }, + "required": [ + "postId", + "kind" + ], + "additionalProperties": false + }, + "note": { + "type": "object", + "indexOnly": true, + "documentsMutable": false, + "canBeDeleted": true, + "indices": [ + { + "name": "byPostBody", + "properties": [ + { + "postId": "asc" + } + ], + "terminal": [ + "$ownerId", + "body" + ] + } + ], + "properties": { + "postId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "contentMediaType": "application/x.dash.dpp.identifier", + "position": 0 + }, + "body": { + "type": "array", + "byteArray": true, + "maxItems": 16, + "position": 1 + } + }, + "required": [ + "postId", + "body" + ], + "additionalProperties": false + } + } +} diff --git a/packages/swift-sdk/Sources/SwiftDashSDK/Core/Utils/DataContractParser.swift b/packages/swift-sdk/Sources/SwiftDashSDK/Core/Utils/DataContractParser.swift index 7fe5a40851e..0c696701c3e 100644 --- a/packages/swift-sdk/Sources/SwiftDashSDK/Core/Utils/DataContractParser.swift +++ b/packages/swift-sdk/Sources/SwiftDashSDK/Core/Utils/DataContractParser.swift @@ -322,8 +322,14 @@ public struct DataContractParser { if let rankedAverageable = indexData["rankedAverageable"] as? Bool { index.rankedAverageable = rankedAverageable } + // A composite terminal is an ordered list of component names; + // the persisted string keeps them joined, in order, so display + // layers show the whole member key. if let terminal = indexData["terminal"] as? String { index.terminal = terminal + } else if let components = indexData["terminal"] as? [String], + !components.isEmpty { + index.terminal = components.joined(separator: " ‖ ") } if let preallocated = indexData["preallocated"] as? Bool { index.preallocated = preallocated