feat(platform)!: composite and flat indexOnly terminals with an entry payload

book/src/drive/index-only-document-types.md

@@ -28,11 +28,14 @@ stores nothing in primary storage. The index entries ARE the rows:
       → Item(<row commitment>, flags)
 ```
 
-The **terminal** — a per-index keyword defaulting to `$ownerId`, or any
-refersTo-typed identifier property (identity, contract, token, permanent
-or deletable document) — is the member key, sitting exactly where a normal non-unique
-index keys by document id; the element is an `Item` instead of a
-`Reference` because there is nothing to point at. The `0` storage marker,
+The **terminal** is the member key, sitting exactly where a normal
+non-unique index keys by document id; the element is an `Item` instead of
+a `Reference` because there is nothing to point at. It is a per-index
+keyword defaulting to `$ownerId`. It may name any schema property a prefix
+position could carry (an identifier with or without a `refersTo`, a
+bounded byte array or string, an integer, a boolean, a date), or an
+ordered **list** of such properties (a *composite* terminal, whose member
+key is their encoded values concatenated). The `0` storage marker,
 value-tree types, and the count/sum/ranked tree derivation are
 byte-identical to the ordinary non-unique layout, which is what lets the
 protocol v14 ranked machinery (see
@@ -41,6 +44,87 @@ types unchanged: "the five most-liked posts in `#dash`" is an
 O(log n + k) read with an O(log n + k) proof, and Items count in
 count/ranked trees exactly as References do.
 
+The member key is the terminal value's **tree-key encoding**, produced by
+the same functions the prefix levels use (the walkers and probes through
+`get_raw_for_document_type`, queries and executed proofs through
+`serialize_value_for_key`, synthesis through `decode_value_for_tree_keys`),
+so nothing about it is specific to a 32-byte identifier: a 33-byte public
+key, a short string or an integer keys the `0` bucket exactly as it would
+key a prefix level, and fee estimation sizes the member key by the
+terminal property's declared bound (`index_only_terminal_max_key_size`)
+rather than by a fixed 32. Structural uniqueness spans the terminal value:
+one entry per (prefix values, terminal value), so two documents by one
+owner that differ only in a scalar terminal are two entries under the
+same prefix.
+
+**Composite terminals.** `"terminal": ["kind", "$ownerId"]` keys the
+member by `encode(kind) ‖ owner`. Every component but the last must be
+fixed width (a byte array with `minItems == maxItems`, an identifier, an
+integer, a boolean, a date), so equality on the leading components is a
+clean key range and synthesis can split the key back; a string can only
+be the last component; the whole key is capped at 255 bytes. Uniqueness
+spans the whole key. Queries bind the components in order: equality
+clauses on the leading ones, then at most one range or `in` clause on the
+next (ordered by it), nothing on the rest. The lowering pads the bound
+prefix with `0xFF` to the key cap for the upper bound of "every key under
+this prefix", and addresses the key itself when the bound component is
+the last one. After equality-bound components are ignored, `orderBy` must
+start at the first remaining component and follow component order without
+gaps, with the same direction for every listed component. A single member-key
+walk cannot sort by a later component alone or mix ascending and descending
+components.
+
+**Flat indexes.** An index with no `properties` at all is *flat*: its
+entries live directly under a level of their own, keyed by a zero byte
+followed by each terminal component name preceded by a zero byte
+(`"\0appEphemeralPubKeyHash\0$ownerId"`), which no property-name tree can
+collide with since property names never contain a zero byte. This level key,
+including its separators, must also fit within 255 bytes:
+
+```text
+[DataContractDocuments, contract_id, 1, <doctype>, "\0<c1>\0<c2>…", 0, <c1 ‖ c2 ‖ …>]
+      → Item(<row commitment> [‖ <entry payload>], flags)
+```
+
+The flat level is registration-time structure, created with the
+property-name trees and kept when the last entry goes (the prune stops at
+its `0` bucket, as on a preallocated index), so every entry costs the same.
+There is no prefix level for an aggregate, a ranking, a time grid, a skip
+trigger or a preallocation to apply to, so a flat index admits none of
+those keywords. A clause-free query on a type with a flat index scans the
+flat level (every other indexOnly type refuses the by-id shape). Non-proof
+responses require this index to cover every property, including optional
+ones, just as filtered queries do; otherwise use a proved projection.
+
+**The entry payload.** `entryPayload: ["walletEphemeralPubKey",
+"encryptedPayload"]` on the document type names top-level properties that
+live in no index: every entry's item carries them after the 32-byte row
+commitment, each length-framed (`u16` big-endian), in property-name order:
+the type's value slot. Byte arrays store their raw bytes and strings store
+UTF-8; other scalars use their tree-key encoding. The length frame preserves
+empty byte arrays and strings without null sentinels, and distinguishes an
+empty string from a NUL string. A payload
+property must be required, scalar and bounded (the sum of the bounds is
+capped by the field value limit), and appears in no index as a property
+or a terminal component. It is still committed (the commitment hashes
+every present property, a payload value through the uncapped payload
+encoding), so the delete probes and the executed-transition verifier keep
+comparing the item's first 32 bytes only, and synthesis decodes the rest
+of the proved element. With more than one index the payload rides in
+every entry; fee estimation sizes the item by the commitment plus the
+payload bound. Together, a flat composite terminal and an entry payload
+make a key-value table:
+
+```json
+"indices": [{ "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] }],
+"entryPayload": ["walletEphemeralPubKey", "encryptedPayload"]
+```
+
+lands at `[…, "\0appEphemeralPubKeyHash\0$ownerId", 0, hash ‖ owner] →
+Item(commitment ‖ len ‖ ciphertext ‖ len ‖ wallet key)`, and a query on
+the hash returns every responder's owner id with the payload decoded off
+the item, as one proof.
+
 **`timeRange` buckets** compose too: a bucketed indexOnly index writes
 one commitment entry per containing bucket under the grid-qualified
 level, exactly as stored types do — the walkers' bucket fan-out, the
@@ -103,7 +187,9 @@ aggregate keywords follow:
 | every non-trigger property appears in ≥ 1 **non-skip** index (prefix or terminal) | only indexed values exist, and a skip index carries no value for trigger-absent documents — covered only there, a property would be validated and committed yet written nowhere |
 | **every index embeds `$ownerId`** (prefix or terminal) | entries are self-authorizing: a delete computed with owner = signer can only ever address the signer's own entries |
 | ≥ 1 index is `$createdAt`-free AND non-`skipIfAbsent` — the **proof index** | executed-transition proofs locate entries from the transition's values alone: they can neither reproduce a block timestamp nor anchor on an entry that may not exist |
-| terminal is `$ownerId` or a single-id refersTo property | the member key must alone be a referable entity id (`identityPublicKey` is compound and rejected) |
+| every terminal component is `$ownerId` or a schema property passing the indexed-shape limits (no arrays or objects; byte arrays ≤ 255 bytes, strings ≤ 63 characters); every component but the last is fixed width; the whole key ≤ 255 bytes | the member key is the components' tree-key encodings concatenated, derived by the same functions the prefix levels use; a leading component must be splittable back and rangeable; grovedb caps keys at 255 bytes; other system properties are refused because the `$createdAt` rules walk the prefix properties |
+| a flat index (no `properties`) admits no countable / summable / ranked / `timeRange` / `skipIfAbsent` / `preallocated` keyword | there is no prefix level for them to apply to |
+| every `entryPayload` property is a required, bounded, top-level scalar in no index | the entry value has no representation for an absent property, estimation sizes the item by the bounds, and a property is either a key or a value |
 | indexed `$createdAt` requires `$createdAt` in `required` | creation only assigns timestamps for required system times |
 | `documentsMutable: false`, no transfers/trading/history/transient | no stored row, no revision |
 | non-unique, non-contested, `nullSearchable` default | v1 scope |

packages/kotlin-sdk/KotlinExampleApp/app/src/main/java/org/dashfoundation/example/ui/contracts/ContractJson.kt

@@ -195,10 +195,17 @@ internal fun indexAxisDescriptors(index: JsonObject): List<String> {
 }
 
 /**
- * The index's member-key property on an indexOnly document type: the
- * declared `terminal`, defaulting to `$ownerId` exactly as DPP
- * normalizes an omitted terminal. `null` on stored (non-indexOnly)
- * document types, where entries are keyed by document id.
+ * The index's member key on an indexOnly document type: the declared
+ * `terminal` (a property name, or the ordered component names of a
+ * composite terminal joined with ` ‖ ` for display), defaulting to
+ * `$ownerId` exactly as DPP normalizes an omitted terminal. `null` on
+ * stored (non-indexOnly) document types, where entries are keyed by
+ * document id.
  */
 internal fun indexTerminal(index: JsonObject, indexOnly: Boolean): String? =
-    index.stringField("terminal") ?: if (indexOnly) "\$ownerId" else null
+    index.stringField("terminal")
+        ?: (index["terminal"] as? JsonArray)
+            ?.mapNotNull { (it as? JsonPrimitive)?.content }
+            ?.takeIf { it.isNotEmpty() }
+            ?.joinToString(" ‖ ")
+        ?: if (indexOnly) "\$ownerId" else null

packages/kotlin-sdk/KotlinExampleApp/app/src/test/java/org/dashfoundation/example/ui/contracts/IndexKeywordDescriptorsTest.kt

@@ -1,6 +1,8 @@
 package org.dashfoundation.example.ui.contracts
 
 import kotlinx.serialization.json.JsonObject
+import kotlinx.serialization.json.add
+import kotlinx.serialization.json.buildJsonArray
 import kotlinx.serialization.json.buildJsonObject
 import kotlinx.serialization.json.put
 import org.junit.Assert.assertEquals
@@ -92,5 +94,10 @@ class IndexKeywordDescriptorsTest {
         val declared = buildJsonObject { put("terminal", "postId") }
         assertEquals("postId", indexTerminal(declared, indexOnly = true))
         assertEquals("postId", indexTerminal(declared, indexOnly = false))
+
+        val composite = buildJsonObject {
+            put("terminal", buildJsonArray { add("postId"); add("\$ownerId") })
+        }
+        assertEquals("postId ‖ \$ownerId", indexTerminal(composite, indexOnly = true))
     }
 }

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -721,10 +721,25 @@
             "description": "Buckets the first index property's timestamp into fixed-length, regularly-spaced (possibly overlapping) time ranges. The window parameters (`range`, `step`, `phase`) are declared in seconds, since a bucket is selected from block time and the target block interval is five seconds; the stored key is the range start as a u64 millisecond timestamp, so it stays directly comparable to the source timestamp it buckets. Enables trending/leaderboard queries within the newest/oldest active range. A system-timestamp source must be listed in the document type's required fields. Several indexes may bucket the same timestamp with different grids — each grid gets its own index subtree, keyed by the property name qualified with the grid parameters. Available from protocol version 14."
           },
           "terminal": {
-            "type": "string",
-            "minLength": 1,
-            "maxLength": 256,
-            "description": "Only on indexOnly document types: names the property whose value is this index entry's member key — the docId-analog terminal key under the index's storage marker, stored as an Item instead of a Reference because there is no primary-storage row. Either \"$ownerId\" (the default when omitted) or an identifier property carrying a refersTo declaration (identity, contract, token, permanentDocument, or deletableDocument). Must not repeat one of the index's listed properties. Available from protocol version 14."
+            "oneOf": [
+              {
+                "type": "string",
+                "minLength": 1,
+                "maxLength": 256
+              },
+              {
+                "type": "array",
+                "items": {
+                  "type": "string",
+                  "minLength": 1,
+                  "maxLength": 256
+                },
+                "minItems": 1,
+                "maxItems": 10,
+                "uniqueItems": true
+              }
+            ],
+            "description": "Only on indexOnly document types: the property, or the ordered list of properties (a composite terminal), whose encoded values, concatenated, are this index entry's member key — the docId-analog terminal key under the index's storage marker, stored as an Item instead of a Reference because there is no primary-storage row. Each component is \"$ownerId\" (the default when the keyword is omitted) or any schema property a prefix position could carry (no arrays or objects; byte arrays of at most 255 bytes, strings of at most 63 characters); every component but the last must be fixed width (a byte array with minItems equal to maxItems, an identifier, an integer, a boolean or a date), and the whole key at most 255 bytes. One entry exists per (prefix values, terminal values). No other system property may be a component, and none may repeat one of the index's listed properties. An index with no properties is a flat index keyed by its terminal alone, admitting no aggregate, ranking, timeRange, skipIfAbsent or preallocated keyword. Available from protocol version 14."
           },
           "preallocated": {
             "type": "boolean",
@@ -736,7 +751,6 @@
           }
         },
         "required": [
-          "properties",
           "name"
         ],
         "dependentRequired": {
@@ -903,6 +917,18 @@
       "type": "boolean",
       "description": "When true, documents of this type are never written to primary storage: the index entries are the rows, each terminating in an Item keyed by the index's `terminal` property instead of a Reference keyed by the document id. Only what is in the indexes exists and is recoverable. Requires: every property required and appearing in at least one index (except a `skipIfAbsent` index's optional first property), $ownerId in at least one index (as a property or terminal), documentsMutable: false, no transfers/trading/history/transient properties, and no doctype-level aggregate keywords (use the index-level count flags). Available from protocol version 14."
     },
+    "entryPayload": {
+      "type": "array",
+      "items": {
+        "type": "string",
+        "minLength": 1,
+        "maxLength": 64
+      },
+      "minItems": 1,
+      "maxItems": 16,
+      "uniqueItems": true,
+      "description": "Only on indexOnly document types: the top-level properties stored in every entry's value, after the 32-byte row commitment, instead of in a key — the type's value slot. Each is length-framed in property-name order and recovered by decoding the proved element: raw bytes for byte arrays, UTF-8 for strings, and tree-key encoding for other scalars. Empty byte arrays and strings remain distinct from null, and an empty string remains distinct from a NUL string. Listed properties must be required, bounded (maxItems on byte arrays, maxLength on strings), scalar, and must not appear in any index as a property or a terminal component; their summed bounds are capped by the field value limit. With more than one index the payload rides in every entry. Fixed when the document type is created. Available from protocol version 14."
+    },
     "actionFees": {
       "type": "object",
       "description": "A fixed fee in credits charged, on top of the gas, for actions on documents of this type, split between the contract's owner pot and its moderators pot (each paid out by a ContractFeeClaim state transition). Whoever pays the gas of the action pays its fee. A transition on a priced action must name the declared owner and moderators amounts, and for feeMultiplier pricing the fee multiplier its signer knew with the increase in percent they accept, in $actionFeeAgreement; it is refused without one (DocumentActionFeeAgreementNotSetError, code 40132), with other amounts or another pricing (code 40133), or when the epoch's multiplier rose beyond the accepted increase (code 40134). At least one action must be priced and a priced action must charge something. Fixed when the document type is published: a contract update cannot add, change or remove the fees of an existing document type. Available from protocol version 14.",

packages/rs-dpp/src/data_contract/document_type/accessors/mod.rs

@@ -945,6 +945,7 @@ impl DocumentTypeV1Getters for DocumentTypeMutRef<'_> {
 /// predate the keyword: V0 and V1 have no field to borrow from, and the
 /// getter hands out a reference.
 static NO_IMMUTABLE_FIELDS: BTreeSet<String> = BTreeSet::new();
+static NO_ENTRY_PAYLOAD: BTreeSet<String> = BTreeSet::new();
 
 impl DocumentTypeV2Getters for DocumentType {
     fn documents_countable(&self) -> bool {
@@ -987,6 +988,15 @@ impl DocumentTypeV2Getters for DocumentType {
         }
     }
 
+    /// The entry-payload property names of an indexOnly type (empty before V2).
+    fn entry_payload(&self) -> &BTreeSet<String> {
+        match self {
+            DocumentType::V0(_) => &NO_ENTRY_PAYLOAD,
+            DocumentType::V1(_) => &NO_ENTRY_PAYLOAD,
+            DocumentType::V2(v2) => v2.entry_payload(),
+        }
+    }
+
     fn documents_can_be_deleted_by_moderators(&self) -> bool {
         match self {
             DocumentType::V0(_) => false,
@@ -1103,6 +1113,15 @@ impl DocumentTypeV2Getters for DocumentTypeRef<'_> {
         }
     }
 
+    /// The entry-payload property names of an indexOnly type (empty before V2).
+    fn entry_payload(&self) -> &BTreeSet<String> {
+        match self {
+            DocumentTypeRef::V0(_) => &NO_ENTRY_PAYLOAD,
+            DocumentTypeRef::V1(_) => &NO_ENTRY_PAYLOAD,
+            DocumentTypeRef::V2(v2) => v2.entry_payload(),
+        }
+    }
+
     fn documents_can_be_deleted_by_moderators(&self) -> bool {
         match self {
             DocumentTypeRef::V0(_) => false,
@@ -1185,6 +1204,15 @@ impl DocumentTypeV2Getters for DocumentTypeMutRef<'_> {
         }
     }
 
+    /// The entry-payload property names of an indexOnly type (empty before V2).
+    fn entry_payload(&self) -> &BTreeSet<String> {
+        match self {
+            DocumentTypeMutRef::V0(_) => &NO_ENTRY_PAYLOAD,
+            DocumentTypeMutRef::V1(_) => &NO_ENTRY_PAYLOAD,
+            DocumentTypeMutRef::V2(v2) => v2.entry_payload(),
+        }
+    }
+
     fn documents_can_be_deleted_by_moderators(&self) -> bool {
         match self {
             DocumentTypeMutRef::V0(_) => false,

packages/rs-dpp/src/data_contract/document_type/accessors/v2/mod.rs

@@ -33,6 +33,10 @@ pub trait DocumentTypeV2Getters {
     /// each terminating in an `Item` keyed by the index's `terminal`
     /// property. Only what is in the indexes exists and is recoverable.
     fn index_only(&self) -> bool;
+    /// On an indexOnly type, the top-level properties stored in every entry's
+    /// value after the row commitment (`entryPayload`), in name order; empty
+    /// elsewhere.
+    fn entry_payload(&self) -> &BTreeSet<String>;
 
     /// Returns whether the contract's moderators may delete documents of this
     /// type (the `canBeDeletedByModerators` keyword, protocol version 14).