diff --git a/plugins/query_engine_plugin/README.md b/plugins/query_engine_plugin/README.md index 6b5bae7b6f..3858011610 100644 --- a/plugins/query_engine_plugin/README.md +++ b/plugins/query_engine_plugin/README.md @@ -45,7 +45,7 @@ for (const fc::variant_object& row : result.rows) { The public interface is `query_result execute(const std::string& query, const std::optional& options = std::nullopt)`. `query_result` carries `schema_version`, -`complete`, `source`, `state`, `columns`, `stats`, and `std::vector rows`. +`complete`, `source`, `state`, `columns`, `stats`, `page`, and `std::vector rows`. Failures throw `query_error`; C++ execution does not create an RPC envelope. `query_options.timeout` is an optional `std::chrono::milliseconds`. Unset, it applies the configured @@ -109,13 +109,15 @@ canonical typed group-key order. LIMIT follows complete grouping, HAVING and ord valid. Exceeding a server cap fails the query; it never silently truncates totals. No JOIN, subquery, UNION, DISTINCT, SQL OFFSET, arbitrary arithmetic/functions, writes or DDL. -C++ pagination uses `query_options.offset`; HTTP params remain exactly `{query}`. +Pagination uses `query_options` in C++ and the `limit` / `offset` / `timeout_ms` params over HTTP +(see [Paging](#paging)). Primary-key equality, leading composite-key prefix and range predicates can narrow capture. Other predicates use a bounded scan and residual evaluation. Secondary-index optimization is not included. ## Values and response -Send a JSON-RPC 2.0 object with method `query.execute` and exactly one named parameter, `query`: +Send a JSON-RPC 2.0 object with method `query.execute` and named parameters: the required `query`, +plus the optional paging members described under [Paging](#paging): ```json {"jsonrpc":"2.0","id":"totals","method":"query.execute","params":{"query":"SELECT SUM(amount) AS total FROM sample.positions"}} @@ -127,8 +129,8 @@ either `result` or `error`. Notifications omit `id` and return HTTP 204 with an invocation failures. Malformed envelopes and unsupported batch arrays return errors with null ID. HTTP limits enforced before dispatch retain the HTTP plugin's normal transport behavior. -Every successful result contains `schema_version`, `complete: true`, `source`, `state`, `columns`, -`rows`, and `stats`. `source.owners` and `state.abis` identify every selected owner. `state.block_id` +Every successful result contains `schema_version` (`1.1`), `complete: true`, `source`, `state`, +`columns`, `rows`, `stats`, and `page`. `source.owners` and `state.abis` identify every selected owner. `state.block_id` identifies the applied state captured by all internal pages; metadata also includes read mode, irreversible height, chain ID, capture time and local sync status. `synced: false` does not prevent reads. A head state may subsequently be forked out; this is a current-state read, not a finality claim. @@ -171,7 +173,66 @@ and the residual predicate always agree with the decoded text. Only the fields a query references are decoded; every other ABI node is skipped, so `COUNT(*)` reads no row bytes and a projected scan is charged for one decoded row at a time. -See [schema/README.md](schema/README.md), the normative schemas, and [examples/request.json](examples/request.json). +### Paging + +Every `query_options` field is available over HTTP as an optional member of `params`: + +| Member | Value | Meaning | +|---|---|---| +| `limit` | integer in [0, 9007199254740991] | Maximum rows in this page; the smaller of it and an SQL LIMIT applies. `0` returns no rows: the columns and `page` are still present. The page is still capped by `query-max-result-rows`, failing with QUERY_LIMIT rather than truncating. | +| `offset` | integer in [0, 9007199254740991] | Rows of the complete, ordered output skipped before the page. An offset beyond the end returns an empty page. | +| `timeout_ms` | integer in [1, `query-timeout-ms`] | A request deadline that may only LOWER the configured one; it still counts from request ingress, queue time included. | + +The members are JSON numbers with an integral value (`2`, `2.0` and `2e0` are all 2), read exactly +from the token and never through a host float; the 2^53-1 bound keeps every value exact in every +JSON client. A string, null, boolean, fractional, negative or out-of-range value, a `timeout_ms` +above the configured `query-timeout-ms` (whose `error.data.limit` names `query-timeout-ms`), any +other member, or a repeated member is INVALID_PARAMS. HTTP cannot extend or disable the configured +deadline; only C++ callers may opt out with `constants::no_deadline`. + +The second two-row page, verbatim from [examples/request-paged.json](examples/request-paged.json): + +```json +{ + "jsonrpc": "2.0", + "id": "positions-page-2", + "method": "query.execute", + "params": { + "query": "SELECT key.id AS id, beneficiary, amount FROM sample.positions ORDER BY amount DESC", + "limit": 2, + "offset": 2, + "timeout_ms": 500 + } +} +``` + +`offset` applies to the complete evaluated output, before any SQL LIMIT: the output is never cut to +the SQL LIMIT first and then skipped into. An SQL LIMIT therefore acts as a cap on the page size, +exactly like the `limit` member — `... LIMIT 1` with `"offset": 1` returns the second output row — and +`page.total_rows` and `page.has_more` always describe the complete output, not an SQL-LIMIT-truncated +one. + +Every result carries `page`, the window it returned over the complete evaluated output: + +| Member | Value | +|---|---| +| `offset` | the requested offset, as an unsigned decimal string | +| `limit` | the effective limit — the smaller of SQL LIMIT and `limit` — as a decimal string, or null when neither is set | +| `returned_rows` | rows in this response; always equal to `stats.returned_rows` | +| `total_rows` | rows of the complete output: rows passing WHERE for a detail query, groups passing HAVING for a grouped one | +| `has_more` | whether output rows remain after this page: `min(offset, total_rows) + returned_rows < total_rows` | + +Each page is a separate request evaluated at its own chain state, so pages taken while the chain +advances can overlap or skip rows; compare `state.block_id` across pages when a consistent view +matters. Pagination never reduces the input evaluated: aggregation, HAVING and ordering always see +every row, and `query-max-scan-rows` still bounds the whole scan. `complete: true` keeps its meaning +— evaluation finished for that state and nothing was silently truncated — while `has_more` says +whether rows exist beyond this window. Paging is how a result larger than `query-max-result-rows` +is read: a single unpaged request for it fails with QUERY_LIMIT. + +See [schema/README.md](schema/README.md), the normative schemas, [examples/request.json](examples/request.json) +and the paged pair [examples/request-paged.json](examples/request-paged.json) / +[examples/response-paged.json](examples/response-paged.json). ## Resource limits @@ -182,20 +243,20 @@ All options are immutable, positive integers available through CLI or config.ini | query-worker-threads | 2 | Query workers; the optional HTTP adapter has the same number of separate workers | | query-max-in-flight | 4 | Engine admissions including retained reads; also the independent HTTP ingress cap | | query-max-query-bytes | 16384 | SQL bytes before ANTLR | -| query-timeout-ms | 1000 | Request deadline, including ingress/queue time; C++ callers may opt out per call | +| query-timeout-ms | 1000 | Request deadline, including ingress/queue time; HTTP `timeout_ms` may only lower it; C++ callers may opt out per call | | query-max-capture-ms | producer_plugin's read-only transaction time | Each chain read callback (the ABI copy and the coherent data capture); unset, the smaller of that time and `query-timeout-ms`; a configured value may not exceed it | | query-max-abi-bytes | 1048576 | ABI blob size per selected owner, checked before copy | | query-max-scan-rows | 100000 | Total candidate rows across owners | | query-max-raw-bytes | 67108864 | Copied ABI, raw rows, continuation and capture overhead | | query-max-memory-bytes | 134217728 | Conservative allocation charges per request; a decoded row's charges are released once it is folded in | | query-max-groups | 10000 | Aggregate groups | -| query-max-result-rows | 10000 | Output rows after offset and SQL/per-call limit | +| query-max-result-rows | 10000 | Output rows of one page, after offset and SQL/per-call limit | | query-max-response-bytes | 8388608 | Encoded success envelope bytes | Additional bounds are 4096 tokens, depth 64, 2048 AST/ABI descriptor nodes, 64 owners and 512 rows per internal page; `error.data.limit` names them `sql-tokens`, `sql-depth`, `sql-nodes`, -`sql-owners`, `abi-depth` and `abi-nodes`, distinct from the option names. Pages do not yield to -another chain state. The capture bound is producer_plugin's read-only transaction time — the +`sql-owners`, `abi-depth` and `abi-nodes`, distinct from the option names. Internal pages do not yield +to another chain state. The capture bound is producer_plugin's read-only transaction time — the smaller of `max-transaction-time` and the effective read-only read window less its minimum — and every check inside a read callback also honors the current read window's own end, so a capture that starts late in a window stops at the window's end instead of running its full budget past it and diff --git a/plugins/query_engine_plugin/examples/request-paged.json b/plugins/query_engine_plugin/examples/request-paged.json new file mode 100644 index 0000000000..6f9644a684 --- /dev/null +++ b/plugins/query_engine_plugin/examples/request-paged.json @@ -0,0 +1,11 @@ +{ + "jsonrpc": "2.0", + "id": "positions-page-2", + "method": "query.execute", + "params": { + "query": "SELECT key.id AS id, beneficiary, amount FROM sample.positions ORDER BY amount DESC", + "limit": 2, + "offset": 2, + "timeout_ms": 500 + } +} diff --git a/plugins/query_engine_plugin/examples/response-paged.json b/plugins/query_engine_plugin/examples/response-paged.json new file mode 100644 index 0000000000..bfaa7b312c --- /dev/null +++ b/plugins/query_engine_plugin/examples/response-paged.json @@ -0,0 +1,80 @@ +{ + "jsonrpc": "2.0", + "id": "positions-page-2", + "result": { + "schema_version": "1.1", + "complete": true, + "source": { + "owners": [ + "sample" + ], + "table": "positions" + }, + "state": { + "chain_id": "1111111111111111111111111111111111111111111111111111111111111111", + "block_id": "2222222222222222222222222222222222222222222222222222222222222222", + "block_num": "42", + "block_time": "2026-09-21T12:00:00.000", + "read_mode": "head", + "last_irreversible_block_num": "40", + "abis": [ + { + "owner": "sample", + "hash": "3333333333333333333333333333333333333333333333333333333333333333" + } + ], + "captured_at": "2026-09-21T12:00:00.050000Z", + "synced": true + }, + "columns": [ + { + "name": "id", + "logical_type": "integer", + "abi_type": "uint64", + "nullable": false, + "encoding": "decimal_string" + }, + { + "name": "beneficiary", + "logical_type": "text", + "abi_type": "name", + "nullable": false, + "encoding": "text" + }, + { + "name": "amount", + "logical_type": "integer", + "abi_type": "int64", + "nullable": false, + "encoding": "decimal_string" + } + ], + "rows": [ + { + "id": "5", + "beneficiary": "bob", + "amount": "40" + }, + { + "id": "3", + "beneficiary": "bob", + "amount": "20" + } + ], + "stats": { + "scanned_rows": "7", + "matched_rows": "7", + "groups": "0", + "returned_rows": "2", + "raw_bytes": "336", + "elapsed_us": "900" + }, + "page": { + "offset": "2", + "limit": "2", + "returned_rows": "2", + "total_rows": "7", + "has_more": true + } + } +} diff --git a/plugins/query_engine_plugin/examples/response.json b/plugins/query_engine_plugin/examples/response.json index eeed16da52..00f6cfbfd9 100644 --- a/plugins/query_engine_plugin/examples/response.json +++ b/plugins/query_engine_plugin/examples/response.json @@ -2,7 +2,7 @@ "jsonrpc": "2.0", "id": "balances-by-beneficiary", "result": { - "schema_version": "1.0", + "schema_version": "1.1", "complete": true, "source": { "owners": [ @@ -63,6 +63,13 @@ "returned_rows": "1", "raw_bytes": "144", "elapsed_us": "1200" + }, + "page": { + "offset": "0", + "limit": "20", + "returned_rows": "1", + "total_rows": "1", + "has_more": false } } } diff --git a/plugins/query_engine_plugin/include/sysio/query_engine_plugin/query.hpp b/plugins/query_engine_plugin/include/sysio/query_engine_plugin/query.hpp index 219fc20ff6..477dfa2301 100644 --- a/plugins/query_engine_plugin/include/sysio/query_engine_plugin/query.hpp +++ b/plugins/query_engine_plugin/include/sysio/query_engine_plugin/query.hpp @@ -8,12 +8,14 @@ #include +#include #include #include #include #include #include #include +#include #include namespace sysio::query_engine { @@ -23,7 +25,7 @@ namespace constants { inline constexpr auto endpoint = "/v1/query/execute"; inline constexpr auto method = "query.execute"; inline constexpr auto version = "2.0"; -inline constexpr auto schema_version = "1.0"; +inline constexpr auto schema_version = "1.1"; inline constexpr auto logger = "query"; inline constexpr uint32_t max_id_characters = 128; inline constexpr int64_t max_numeric_id = 4294967295LL; @@ -40,8 +42,24 @@ inline constexpr uint32_t decimal_base = 10; inline constexpr auto value_namespace = "value"; inline constexpr auto key_namespace = "key"; inline constexpr size_t max_owners = 64; +/// Smallest request `limit` and `offset`. +inline constexpr uint64_t min_request_integer = 0; +/// Largest request integer (`limit`, `offset`, `timeout_ms`): 2^53-1, exactly representable by every JSON client. +inline constexpr uint64_t max_request_integer = 9007199254740991ULL; +/// Smallest request `timeout_ms`; zero would expire before admission. +inline constexpr uint64_t min_request_timeout_ms = 1; } // namespace constants +/// Member names of the `query.execute` params object: one spelling for parsing, errors and documentation. +namespace request_field { +inline constexpr auto query = "query"; +inline constexpr auto limit = "limit"; +inline constexpr auto offset = "offset"; +inline constexpr auto timeout_ms = "timeout_ms"; +/// Every member params may carry; any other member is INVALID_PARAMS. +inline constexpr auto all = std::to_array({query, limit, offset, timeout_ms}); +} // namespace request_field + /// Names reported in `error.data.limit` for the fixed parser/ABI bounds and for the chain's read /// window. None is a query-plugin option, so none carries the `query-` prefix the options use. namespace bound { @@ -64,6 +82,10 @@ inline constexpr auto state = "state"; inline constexpr auto columns = "columns"; inline constexpr auto rows = "rows"; inline constexpr auto stats = "stats"; +inline constexpr auto page = "page"; +inline constexpr auto offset = "offset"; +inline constexpr auto total_rows = "total_rows"; +inline constexpr auto has_more = "has_more"; inline constexpr auto owners = "owners"; inline constexpr auto table = "table"; inline constexpr auto name = "name"; @@ -235,6 +257,11 @@ class query_budget { void assert_limit(uint64_t value, uint64_t maximum, const char* limit) const; /// Monotonic elapsed time used for result stats and deterministic tests. uint64_t elapsed_us() const; + /// Apply validated per-request options after the envelope was parsed with this budget: `limit` and + /// `offset` replace the current values, and a timeout only ever moves the deadline EARLIER, measured + /// from `started`, so ingress and queue time stay inside it; `options.timeout` follows the lowered + /// deadline. A negative timeout is INVALID_PARAMS. + void apply_request_options(const query_options& requested); query_config config; query_options options; now_function now; @@ -255,6 +282,11 @@ class query_budget { uint64_t accounted_bytes = 0; /// Highest accounted_bytes reached, unaffected by releases. uint64_t peak_accounted_bytes = 0; + +private: + /// The absolute deadline `timeout` after `started`: no deadline for constants::no_deadline, and + /// INVALID_PARAMS for a negative timeout or one beyond the monotonic clock's range. + clock::time_point deadline_for(std::chrono::milliseconds timeout) const; }; /// Exact, checked integer arithmetic; no host floating-point conversion is permitted. @@ -469,12 +501,17 @@ struct query_result { std::vector columns; std::vector rows; fc::variant_object stats; + /// Window over the complete ordered output: requested `offset`, effective `limit` (the smaller of SQL + /// LIMIT and the per-call limit, null when neither is set), `returned_rows`, `total_rows` and `has_more`. + fc::variant_object page; }; /// Parsed JSON-RPC request. A missing ID is a notification; null is a valid response ID. struct query_request { fc::variant id; bool notification = false; std::string query; + /// `limit`, `offset` and `timeout_ms` from params; the timeout never exceeds `query-timeout-ms`. + query_options options; std::optional invocation_error; }; @@ -552,4 +589,4 @@ query_request parse_request(std::string_view, query_budget&); } // namespace sysio::query_engine -FC_REFLECT(sysio::query_engine::query_result, (schema_version)(complete)(source)(state)(columns)(rows)(stats)) +FC_REFLECT(sysio::query_engine::query_result, (schema_version)(complete)(source)(state)(columns)(rows)(stats)(page)) diff --git a/plugins/query_engine_plugin/schema/README.md b/plugins/query_engine_plugin/schema/README.md index ada0d49b7e..4b9e64953b 100644 --- a/plugins/query_engine_plugin/schema/README.md +++ b/plugins/query_engine_plugin/schema/README.md @@ -1,12 +1,22 @@ # query.execute schemas Both schemas use JSON Schema Draft-04 and are validated with the repository's RapidJSON validator. -The response contract is version `1.0`. Breaking shape or encoding changes require a new version. +The response contract is version `1.1`. Shape or encoding changes require a new version. The request schema describes the single-request profile; method validation and SQL semantics occur in the service. Notifications have no response document (HTTP 204, empty body). -Success and error envelopes are mutually exclusive. A successful result is always complete for the -submitted SQL, including its explicit LIMIT. Budget/decode failures contain only an error. +Request `params` hold the required `query` and the optional paging members `limit` and `offset` +(integers in [0, 9007199254740991]) and `timeout_ms` (an integer in [1, 9007199254740991]). The +schema bounds `timeout_ms` only by that range; the service additionally rejects a value above the +configured `query-timeout-ms` as INVALID_PARAMS, because a request may only lower the deadline. +Request integers are exact JSON numbers; response counters are unsigned decimal strings. + +Success and error envelopes are mutually exclusive. A successful result is never silently truncated: +`page` says which window of the complete evaluated output it holds. Budget/decode failures contain +only an error. `page` carries the requested `offset`, the effective `limit` (the smaller of SQL LIMIT +and the request limit, or null), `returned_rows` (equal to `stats.returned_rows`), `total_rows` (rows +passing WHERE, or groups passing HAVING) and `has_more`, which is true exactly when +`min(offset, total_rows) + returned_rows < total_rows`. `source.owners` lists distinct contract accounts in chain-name order. `state.abis` contains the matching owner/hash entries in the same order. Every row was captured at `state.block_id` in one @@ -21,7 +31,8 @@ decimal-string `amount`, text `symbol`, unsigned-decimal-string `precision`; ext the same string rule. A variant projects as the ABI type name and its normalized value. Integration tests additionally verify unique column names, identical row/column key sets, -nullability, encoding, asset members, owner/ABI correspondence, and returned-row counts. These +nullability, encoding, asset members, owner/ABI correspondence, returned-row counts, and the +agreement of `page` with the rows and `stats` it describes. These relationships depend on runtime columns and cannot be expressed by this static Draft-04 schema. Numeric RPC IDs and error positions/codes are explicit exceptions to string-only numeric cells. diff --git a/plugins/query_engine_plugin/schema/query-request.schema.json b/plugins/query_engine_plugin/schema/query-request.schema.json index 90491dac0c..523f351861 100644 --- a/plugins/query_engine_plugin/schema/query-request.schema.json +++ b/plugins/query_engine_plugin/schema/query-request.schema.json @@ -6,6 +6,14 @@ "params" ], "additionalProperties": false, + "definitions": { + "requestInteger": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991, + "description": "An exact JSON integer in [0, 2^53-1], the range every JSON client represents exactly." + } + }, "properties": { "jsonrpc": { "enum": [ @@ -38,10 +46,27 @@ "type": "string", "minLength": 1, "description": "UTF-8 SQL; runtime enforces the configured byte limit (default 16384)." + }, + "limit": { + "$ref": "#/definitions/requestInteger", + "description": "Maximum rows in this page, combined with an SQL LIMIT by taking the smaller; 0 returns no rows. A page above query-max-result-rows fails with QUERY_LIMIT." + }, + "offset": { + "$ref": "#/definitions/requestInteger", + "description": "Rows of the complete, ordered output skipped before the page; beyond the end yields an empty page." + }, + "timeout_ms": { + "allOf": [ + { + "$ref": "#/definitions/requestInteger" + } + ], + "minimum": 1, + "description": "Request deadline in milliseconds, counted from ingress; may only lower the configured query-timeout-ms (a larger value is INVALID_PARAMS)." } } } }, "$schema": "http://json-schema.org/draft-04/schema#", - "title": "WIRE query.execute single-request profile v1" + "title": "WIRE query.execute single-request profile v1.1" } diff --git a/plugins/query_engine_plugin/schema/query-response.schema.json b/plugins/query_engine_plugin/schema/query-response.schema.json index 89206c16d7..6d2fe911ce 100644 --- a/plugins/query_engine_plugin/schema/query-response.schema.json +++ b/plugins/query_engine_plugin/schema/query-response.schema.json @@ -1,6 +1,6 @@ { "$schema": "http://json-schema.org/draft-04/schema#", - "title": "WIRE query.execute response v1", + "title": "WIRE query.execute response v1.1", "oneOf": [ { "type": "object", @@ -266,6 +266,41 @@ } } }, + "page": { + "type": "object", + "required": [ + "offset", + "limit", + "returned_rows", + "total_rows", + "has_more" + ], + "additionalProperties": false, + "properties": { + "offset": { + "$ref": "#/definitions/unsignedDecimal" + }, + "limit": { + "oneOf": [ + { + "$ref": "#/definitions/unsignedDecimal" + }, + { + "type": "null" + } + ] + }, + "returned_rows": { + "$ref": "#/definitions/unsignedDecimal" + }, + "total_rows": { + "$ref": "#/definitions/unsignedDecimal" + }, + "has_more": { + "type": "boolean" + } + } + }, "error": { "type": "object", "required": [ @@ -363,13 +398,14 @@ "state", "columns", "rows", - "stats" + "stats", + "page" ], "additionalProperties": false, "properties": { "schema_version": { "enum": [ - "1.0" + "1.1" ] }, "complete": { @@ -401,6 +437,9 @@ }, "stats": { "$ref": "#/definitions/stats" + }, + "page": { + "$ref": "#/definitions/page" } } } diff --git a/plugins/query_engine_plugin/src/query_config.cpp b/plugins/query_engine_plugin/src/query_config.cpp index e0fe97bbfb..2330355682 100644 --- a/plugins/query_engine_plugin/src/query_config.cpp +++ b/plugins/query_engine_plugin/src/query_config.cpp @@ -7,6 +7,9 @@ namespace sysio::query_engine { namespace { +constexpr auto negative_timeout_message = "Timeout must be a nonnegative millisecond count"; +constexpr auto clock_range_message = "Timeout exceeds the monotonic clock range"; + /// Parse decimal option text without accepting a leading minus through unsigned wraparound. template T option_value(const boost::program_options::variables_map& options, const char* name, T fallback) { @@ -44,7 +47,8 @@ void add_options(boost::program_options::options_description& options) { options.add_options()( option::timeout_ms, boost::program_options::value()->default_value(std::to_string(defaults::timeout.count())), - "Query deadline in milliseconds, including queue time; C++ callers may opt out per call"); + "Query deadline in milliseconds, including queue time; HTTP timeout_ms may only lower it; C++ callers may " + "opt out per call"); // No default: producer_plugin's read-only transaction budget governs unless an operator sets a // tighter capture bound, and a configured value may never exceed that budget. options.add_options()(option::max_capture_ms, boost::program_options::value(), @@ -133,21 +137,22 @@ query_budget::query_budget(query_config config, now_function now, const std::opt , deadline(clock::time_point::max()) { config.validate(); // An unset per-call timeout applies the configured deadline; only constants::no_deadline opts out. - const auto timeout = this->options.timeout.value_or(config.timeout); + deadline = deadline_for(this->options.timeout.value_or(config.timeout)); +} + +query_budget::clock::time_point query_budget::deadline_for(std::chrono::milliseconds timeout) const { + // A negative timeout exceeded no limit; it is simply not a duration, so no option is named. if (timeout < std::chrono::milliseconds::zero()) - throw query_error(error_kind::INVALID_PARAMS, "Timeout must be a nonnegative millisecond count", std::nullopt, - option::timeout_ms); + throw query_error(error_kind::INVALID_PARAMS, negative_timeout_message); if (timeout == constants::no_deadline) - return; + return clock::time_point::max(); const auto maximum = std::chrono::duration_cast(clock::duration::max()); if (timeout > maximum) - throw query_error(error_kind::INVALID_PARAMS, "Timeout exceeds the monotonic clock range", std::nullopt, - option::timeout_ms); + throw query_error(error_kind::INVALID_PARAMS, clock_range_message, std::nullopt, option::timeout_ms); const auto duration = std::chrono::duration_cast(timeout); - if (started > deadline - duration) - throw query_error(error_kind::INVALID_PARAMS, "Timeout exceeds the monotonic clock range", std::nullopt, - option::timeout_ms); - deadline = started + duration; + if (started > clock::time_point::max() - duration) + throw query_error(error_kind::INVALID_PARAMS, clock_range_message, std::nullopt, option::timeout_ms); + return started + duration; } void query_budget::check() const { @@ -216,4 +221,17 @@ uint64_t query_budget::elapsed_us() const { return std::chrono::duration_cast(now() - started).count(); } +void query_budget::apply_request_options(const query_options& requested) { + // Compared as durations: constants::no_deadline would overflow a time point. + const auto lowered = requested.timeout && *requested.timeout < options.timeout.value_or(config.timeout); + // Validated before any mutation, so a rejected request leaves the budget as it was. + const auto lowered_deadline = lowered ? deadline_for(*requested.timeout) : deadline; + options.limit = requested.limit; + options.offset = requested.offset; + if (lowered) { + deadline = lowered_deadline; + options.timeout = requested.timeout; + } +} + } // namespace sysio::query_engine diff --git a/plugins/query_engine_plugin/src/query_executor.cpp b/plugins/query_engine_plugin/src/query_executor.cpp index 3fa36f4ff6..59781e475f 100644 --- a/plugins/query_engine_plugin/src/query_executor.cpp +++ b/plugins/query_engine_plugin/src/query_executor.cpp @@ -230,6 +230,14 @@ projected_row project(const typed_plan& plan, const std::vector& fields, } return result; } + +/// The effective window size: the smaller of an explicit SQL LIMIT and the per-call limit, or none +/// when neither is set. +std::optional combined_limit(const std::optional& sql, const std::optional& call) { + if (sql && call) + return std::min(*sql, *call); + return sql ? sql : call; +} } // namespace int compare_keys(const std::vector& left, const std::vector& right) { @@ -317,9 +325,10 @@ query_result evaluate(const typed_plan& plan, captured_input input, query_budget return left.owner < right.owner; return compare_keys(left.primary_key, right.primary_key) < 0; }); - const auto offset = std::min(output.size(), budget.options.offset); - const auto count = std::min({uint64_t(output.size()) - offset, plan.ast.limit.value_or(output.size()), - budget.options.limit.value_or(output.size())}); + const uint64_t total = output.size(); + const auto offset = std::min(total, budget.options.offset); + const auto window_limit = combined_limit(plan.ast.limit, budget.options.limit); + const auto count = std::min(total - offset, window_limit.value_or(total)); budget.assert_limit(count, budget.config.max_result_rows, option::max_result_rows); budget.charge_memory(count * sizeof(fc::variant_object) * allocation_factor + plan.columns.size() * sizeof(output_column)); @@ -350,12 +359,17 @@ query_result evaluate(const typed_plan& plan, captured_input input, query_budget response_field::returned_rows, std::to_string(count))( response_field::raw_bytes, std::to_string(budget.raw_bytes))(response_field::elapsed_us, std::to_string(budget.elapsed_us())); + auto page = fc::mutable_variant_object()(response_field::offset, std::to_string(budget.options.offset))( + response_field::limit, window_limit ? fc::variant(std::to_string(*window_limit)) + : fc::variant())(response_field::returned_rows, std::to_string(count))( + response_field::total_rows, std::to_string(total))(response_field::has_more, offset + count < total); budget.check(); return {.source = std::move(source), .state = input.state.get_object(), .columns = std::move(columns), .rows = std::move(rows), - .stats = std::move(stats)}; + .stats = std::move(stats), + .page = std::move(page)}; } catch (const std::overflow_error&) { throw query_error(error_kind::VALUE_ERROR, "Aggregate overflow"); } diff --git a/plugins/query_engine_plugin/src/query_http_handler.cpp b/plugins/query_engine_plugin/src/query_http_handler.cpp index 0a47fdd982..0936e5261f 100644 --- a/plugins/query_engine_plugin/src/query_http_handler.cpp +++ b/plugins/query_engine_plugin/src/query_http_handler.cpp @@ -150,6 +150,7 @@ struct query_http_handler::impl { } /// Bound ingress independently of engine admission, including queued HTTP bodies and response buffers. + /// The request's own `limit`, `offset` and `timeout_ms` are applied to the budget before admission. void submit(std::string body, url_response_callback callback) { auto budget = engine->create_budget(query_options{.timeout = engine->config().timeout}); query_request parsed; @@ -163,6 +164,8 @@ struct query_http_handler::impl { respond(&parsed, std::move(callback), create_error(&parsed, *parsed.invocation_error)); return; } + // Cannot throw: timeout_ms is at most the configured timeout, whose deadline this budget already holds. + budget->apply_request_options(parsed.options); auto task = std::make_shared(); task->request = std::move(parsed); task->budget = std::move(budget); diff --git a/plugins/query_engine_plugin/src/query_rpc.cpp b/plugins/query_engine_plugin/src/query_rpc.cpp index b77ca347a3..b3ccbf925c 100644 --- a/plugins/query_engine_plugin/src/query_rpc.cpp +++ b/plugins/query_engine_plugin/src/query_rpc.cpp @@ -1,13 +1,19 @@ #include #include +#include #include #include #include #include +#include +#include #include +#include +#include #include +#include namespace sysio::query_engine { namespace { @@ -17,41 +23,76 @@ constexpr auto key_version = "jsonrpc"; constexpr auto key_id = "id"; constexpr auto key_method = "method"; constexpr auto key_params = "params"; -constexpr auto key_query = "query"; constexpr auto invalid_envelope_message = "Invalid JSON-RPC request"; constexpr auto invalid_params_message = "Expected named query parameter"; +constexpr auto invalid_parameter_name_message = "Unknown query parameter"; +constexpr auto duplicate_parameter_message = "Duplicate query parameter"; +constexpr auto integer_range_format = "{} must be an integer in [{}, {}]"; +constexpr auto lowers_only_format = "{} may only lower {}"; +/// Nesting depth of the root object's members. +constexpr uint32_t root_depth = 1; +/// Nesting depth of the members of params. +constexpr uint32_t params_depth = 2; +/// Every member of the root request object. +constexpr auto envelope_members = std::to_array({key_version, key_id, key_method, key_params}); constexpr uint32_t parse_flags = rapidjson::kParseValidateEncodingFlag | rapidjson::kParseNumbersAsStringsFlag | rapidjson::kParseIterativeFlag; -/// Preflight bounds JSON nesting and retains token kinds without ever parsing a host float. +/// The JSON token a params member's value was written as; `other` is anything else, or an absent member. +enum class token_kind : uint8_t { other, string, number }; + +/// Preflight bounds JSON nesting and retains token kinds without ever parsing a host float. Under +/// kParseNumbersAsStringsFlag the document holds numbers as strings, so the token kind of every +/// request_field member of params is recorded here. struct envelope_preflight : rapidjson::BaseReaderHandler, envelope_preflight> { query_budget& budget; uint32_t depth = 0; std::string root_key; - std::string parameter_key; - bool version_string = false, method_string = false, id_number = false, query_string = false; + bool version_string = false, method_string = false, id_number = false; + /// Value token kinds of the request_field::all members of params, in that order. + std::array parameter_kinds{}; + /// Slot in parameter_kinds of the params member being read; empty for any other name. + std::optional parameter; explicit envelope_preflight(query_budget& budget) : budget(budget) {} + /// The slot of `name` in request_field::all. + static std::optional slot(std::string_view name) { + const auto found = std::ranges::find(request_field::all, name); + if (found == request_field::all.end()) + return std::nullopt; + return static_cast(std::distance(request_field::all.begin(), found)); + } + /// The recorded value token kind of the params member `name`. + token_kind kind(std::string_view name) const { + const std::optional index = slot(name); + return index ? parameter_kinds[*index] : token_kind::other; + } + /// Record the kind of a value written directly under a request_field member of params. + void record(token_kind token) { + if (depth == params_depth && parameter) + parameter_kinds[*parameter] = token; + } bool Key(const char* text, rapidjson::SizeType length, bool) { budget.check(); - if (depth == 1) + if (depth == root_depth) { root_key.assign(text, length); - else if (depth == 2 && root_key == key_params) - parameter_key.assign(text, length); + parameter.reset(); + } else if (depth == params_depth && root_key == key_params) + parameter = slot(std::string_view(text, length)); return true; } bool String(const char*, rapidjson::SizeType, bool) { - if (depth == 1 && root_key == key_version) + if (depth == root_depth && root_key == key_version) version_string = true; - if (depth == 1 && root_key == key_method) + if (depth == root_depth && root_key == key_method) method_string = true; - if (depth == 2 && root_key == key_params && parameter_key == key_query) - query_string = true; + record(token_kind::string); return true; } bool RawNumber(const char*, rapidjson::SizeType, bool) { - if (depth == 1 && root_key == key_id) + if (depth == root_depth && root_key == key_id) id_number = true; + record(token_kind::number); return true; } bool StartObject() { @@ -69,6 +110,28 @@ struct envelope_preflight : rapidjson::BaseReaderHandler, enve } }; +/// Preserve embedded NUL bytes so validation never accepts a truncated spelling. +std::string_view to_string_view(const rapidjson::Value& value) { + return {value.GetString(), value.GetStringLength()}; +} + +/// Why an object's member names fail a closed member list. +enum class member_violation { unknown, duplicate }; + +/// The first member of `object` whose name is not in `allowed` or repeats an earlier name. +std::optional find_member_violation(const rapidjson::Value& object, + std::span allowed) { + std::set names; + for (auto member = object.MemberBegin(); member != object.MemberEnd(); ++member) { + const std::string_view name = to_string_view(member->name); + if (std::find(allowed.begin(), allowed.end(), name) == allowed.end()) + return member_violation::unknown; + if (!names.insert(name).second) + return member_violation::duplicate; + } + return std::nullopt; +} + /// Count validated UTF-8 code points, the JSON Schema maxLength unit. size_t characters(std::string_view text) { size_t result = 0; @@ -79,10 +142,18 @@ size_t characters(std::string_view text) { return result; } -/// Accept integral JSON numbers, including exponent notation, with exact lexical arithmetic. -int64_t parse_id(std::string_view number) { - const bool negative = number.starts_with('-'); - if (negative) +/// The exact integer value of a raw JSON number token. +struct integral_value { + bool negative = false; + uint64_t magnitude = 0; +}; + +/// Exact lexical integer of a raw JSON number token: the fraction and exponent are applied to the +/// decimal digits, never through a host float, so `42.0` and `4.2e1` are 42 while `0.1` is not an +/// integer. Empty when the token is not integral or its magnitude exceeds `magnitude_bound`. +std::optional parse_integral(std::string_view number, uint64_t magnitude_bound) { + integral_value result{.negative = number.starts_with('-')}; + if (result.negative) number.remove_prefix(1); const auto exponent_start = number.find_first_of("eE"); int32_t exponent = 0; @@ -92,7 +163,7 @@ int64_t parse_id(std::string_view number) { text.remove_prefix(1); const auto parsed = std::from_chars(text.data(), text.data() + text.size(), exponent); if (parsed.ec != std::errc{} || parsed.ptr != text.data() + text.size()) - throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); + return std::nullopt; number = number.substr(0, exponent_start); } std::string digits; @@ -103,28 +174,96 @@ int64_t parse_id(std::string_view number) { digits.push_back(character); const auto first = digits.find_first_not_of('0'); if (first == std::string::npos) - return 0; + return result; digits.erase(0, first); scale -= exponent; while (scale > 0 && !digits.empty() && digits.back() == '0') { digits.pop_back(); --scale; } - constexpr size_t max_digits = 10; + const auto max_digits = std::to_string(magnitude_bound).size(); if (scale > 0 || scale < -int64_t(max_digits) || digits.size() + size_t(-scale) > max_digits) - throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); + return std::nullopt; digits.append(size_t(-scale), '0'); - int64_t result = 0; - for (char character : digits) - result = result * constants::decimal_base + character - '0'; - if (result > constants::max_numeric_id) + for (char character : digits) { + const uint64_t digit = character - '0'; + // A bound near UINT64_MAX admits 20-digit tokens, which would otherwise wrap. + if (result.magnitude > (std::numeric_limits::max() - digit) / constants::decimal_base) + return std::nullopt; + result.magnitude = result.magnitude * constants::decimal_base + digit; + } + if (result.magnitude > magnitude_bound) + return std::nullopt; + return result; +} + +/// Accept integral JSON numbers, including exponent notation, within the numeric ID range. +int64_t parse_id(std::string_view number) { + const auto id = parse_integral(number, constants::max_numeric_id); + if (!id) throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); - return negative ? -result : result; + const auto magnitude = static_cast(id->magnitude); + return id->negative ? -magnitude : magnitude; } -/// Preserve embedded NUL bytes so validation never accepts a truncated spelling. -std::string_view string_view(const rapidjson::Value& value) { - return {value.GetString(), value.GetStringLength()}; +/// An optional integer member of params and the inclusive range its exact value must lie in. +struct integer_parameter { + std::string_view field; + uint64_t minimum; + uint64_t maximum; +}; +constexpr integer_parameter limit_parameter{request_field::limit, constants::min_request_integer, + constants::max_request_integer}; +constexpr integer_parameter offset_parameter{request_field::offset, constants::min_request_integer, + constants::max_request_integer}; +constexpr integer_parameter timeout_parameter{request_field::timeout_ms, constants::min_request_timeout_ms, + constants::max_request_integer}; + +/// One optional integer member of params: absent, or a raw JSON number token (never a string, null +/// or boolean) whose exact value lies in the parameter's range; anything else is INVALID_PARAMS. +std::optional parse_integer_parameter(const rapidjson::Value& params, const envelope_preflight& preflight, + const integer_parameter& parameter) { + // A Value key carries its length; a bare StringRef converts to const char* and FindMember would strlen it. + const rapidjson::Value name(rapidjson::StringRef(parameter.field.data(), parameter.field.size())); + const auto member = params.FindMember(name); + if (member == params.MemberEnd()) + return std::nullopt; + const std::optional value = + preflight.kind(parameter.field) == token_kind::number && member->value.IsString() + ? parse_integral(to_string_view(member->value), parameter.maximum) + : std::nullopt; + if (!value || (value->negative && value->magnitude) || value->magnitude < parameter.minimum) + throw query_error(error_kind::INVALID_PARAMS, + fmt::format(integer_range_format, parameter.field, parameter.minimum, parameter.maximum)); + return value->magnitude; +} + +/// Validate params: only the request_field members, each at most once; `query` a non-empty string; +/// `limit` and `offset` integers in [0, 2^53-1]; `timeout_ms` an integer of at least one that may only +/// lower the configured `query-timeout-ms`. Every failure throws INVALID_PARAMS. +query_options parse_parameters(const rapidjson::Value& params, const envelope_preflight& preflight, + const query_config& config) { + if (!params.IsObject()) + throw query_error(error_kind::INVALID_PARAMS, invalid_params_message); + if (const auto violation = find_member_violation(params, request_field::all)) + throw query_error(error_kind::INVALID_PARAMS, *violation == member_violation::unknown + ? invalid_parameter_name_message + : duplicate_parameter_message); + const auto query = params.FindMember(request_field::query); + if (query == params.MemberEnd() || preflight.kind(request_field::query) != token_kind::string || + !query->value.IsString() || query->value.GetStringLength() == 0) + throw query_error(error_kind::INVALID_PARAMS, invalid_params_message); + query_options options; + options.limit = parse_integer_parameter(params, preflight, limit_parameter); + options.offset = parse_integer_parameter(params, preflight, offset_parameter).value_or(options.offset); + if (const auto milliseconds = parse_integer_parameter(params, preflight, timeout_parameter)) { + if (*milliseconds > static_cast(config.timeout.count())) + throw query_error(error_kind::INVALID_PARAMS, + fmt::format(lowers_only_format, request_field::timeout_ms, option::timeout_ms), + std::nullopt, option::timeout_ms); + options.timeout = std::chrono::milliseconds(*milliseconds); + } + return options; } } // namespace @@ -143,15 +282,10 @@ query_request parse_request(std::string_view body, query_budget& budget) { throw query_error(error_kind::PARSE_ERROR, "Invalid JSON"); if (!document.IsObject()) throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); - std::set members; - for (auto member = document.MemberBegin(); member != document.MemberEnd(); ++member) { - const std::string name(member->name.GetString(), member->name.GetStringLength()); - if ((name != key_version && name != key_id && name != key_method && name != key_params) || - !members.insert(name).second) - throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); - } + if (find_member_violation(document, envelope_members)) + throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); if (!document.HasMember(key_version) || !preflight.version_string || !document[key_version].IsString() || - string_view(document[key_version]) != constants::version || !document.HasMember(key_method) || + to_string_view(document[key_version]) != constants::version || !document.HasMember(key_method) || !preflight.method_string || !document[key_method].IsString()) throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); query_request request; @@ -161,23 +295,29 @@ query_request parse_request(std::string_view body, query_budget& budget) { if (id.IsNull()) request.id = fc::variant(); else if (preflight.id_number && id.IsString()) - request.id = parse_id(string_view(id)); - else if (id.IsString() && characters(string_view(id)) <= constants::max_id_characters) - request.id = std::string(string_view(id)); + request.id = parse_id(to_string_view(id)); + else if (id.IsString() && characters(to_string_view(id)) <= constants::max_id_characters) + request.id = std::string(to_string_view(id)); else throw query_error(error_kind::INVALID_REQUEST, invalid_envelope_message); } - if (string_view(document[key_method]) != constants::method) { + if (to_string_view(document[key_method]) != constants::method) { request.invocation_error.emplace(error_kind::METHOD_NOT_FOUND, "Method not found"); return request; } - if (!document.HasMember(key_params) || !document[key_params].IsObject() || document[key_params].MemberCount() != 1 || - !document[key_params].HasMember(key_query) || !preflight.query_string || - !document[key_params][key_query].IsString() || document[key_params][key_query].GetStringLength() == 0) { + if (!document.HasMember(key_params)) { request.invocation_error.emplace(error_kind::INVALID_PARAMS, invalid_params_message); return request; } - const auto sql = string_view(document[key_params][key_query]); + const auto& params = document[key_params]; + // A params failure is an invocation error: the envelope is valid, so the response echoes its ID. + try { + request.options = parse_parameters(params, preflight, budget.config); + } catch (const query_error& error) { + request.invocation_error = error; + return request; + } + const auto sql = to_string_view(params[request_field::query]); if (sql.size() > budget.config.max_query_bytes) { request.invocation_error.emplace(error_kind::QUERY_LIMIT, "Query resource limit exceeded", std::nullopt, option::max_query_bytes); diff --git a/plugins/query_engine_plugin/test/README.md b/plugins/query_engine_plugin/test/README.md index bc3c23f770..f4e60b8ac9 100644 --- a/plugins/query_engine_plugin/test/README.md +++ b/plugins/query_engine_plugin/test/README.md @@ -63,7 +63,14 @@ handler with requests in flight and re-queue a read the window cut short, restor charges (rows and bytes against an uncut reference run, every counter at unit level), naming the window in a later deadline failure over both the C++ and the HTTP path until a retry succeeds, and never retrying a capture that overran its own budget; `query_execute` pins the caller-owned budget -contract (required, single use). Integration tests also pin that a text extreme, +contract (required, single use), the `page` window for detail and grouped queries (offset beyond the +end, `limit` 0, the smaller of SQL LIMIT and the per-call limit) and request options that only ever +move a deadline earlier. `query_rpc` pins the HTTP paging params (`limit`, `offset`, `timeout_ms` +as exact JSON integers within their bounds, a `timeout_ms` above `query-timeout-ms`, unknown and +duplicate members), and the schema suite validates the paged examples and the new members' negative +cases. The application tests page over HTTP past `query-max-result-rows` (an unpaged request still +fails with QUERY_LIMIT) and show a lowered `timeout_ms` timing out while the configured deadline +does not. Integration tests also pin that a text extreme, finalized aggregates and the engine's peak statistic track live state. The application tests pin the read-exclusive queue (a write window longer than the deadline times a query out), the capture budget derived from the read window, and that the plugin binds the window on every read (the capture diff --git a/plugins/query_engine_plugin/test/query_fixture.hpp b/plugins/query_engine_plugin/test/query_fixture.hpp index b698093f41..9d29fd13ef 100644 --- a/plugins/query_engine_plugin/test/query_fixture.hpp +++ b/plugins/query_engine_plugin/test/query_fixture.hpp @@ -129,13 +129,6 @@ inline std::string flag(const char* option) { return std::string("--") + option; } -/// Production JSON-RPC envelope without a second route implementation. -inline std::string request_body(const std::string& sql) { - return fc::json::to_string(fc::mutable_variant_object()("jsonrpc", constants::version)("id", "test")( - "method", constants::method)("params", fc::mutable_variant_object()("query", sql)), - fc::time_point::maximum()); -} - /// Run the production HTTP handler through its worker queues, with a finite test harness deadline. inline fc::variant submit(query_http_handler& handler, read_queue& reads, const std::string& sql) { std::promise completion; diff --git a/plugins/query_engine_plugin/test/query_schema.hpp b/plugins/query_engine_plugin/test/query_schema.hpp index 453e05a855..fa866a3441 100644 --- a/plugins/query_engine_plugin/test/query_schema.hpp +++ b/plugins/query_engine_plugin/test/query_schema.hpp @@ -1,6 +1,7 @@ #pragma once #include #include +#include #include @@ -8,11 +9,30 @@ #include #include +#include #include +#include #include #include +#include namespace sysio::query_engine::test { +/// The `id` every test request envelope carries. +inline constexpr auto request_id = "test"; +/// A query over the fixture's wide table, for envelope tests that never execute it. +inline constexpr auto wide_select_sql = "SELECT * FROM sample.wide"; + +/// The production `query.execute` envelope whose params carry `query` plus `members`, each written +/// `,"name":value` and spelled verbatim so number tokens reach the parser, route or validator exactly +/// as written. No second route implementation is involved. +inline std::string request_body(const std::string& sql, const std::string& members = {}) { + const auto json = [](const std::string& text) { + return fc::json::to_string(fc::variant(text), fc::time_point::maximum()); + }; + return R"({"jsonrpc":)" + json(constants::version) + R"(,"id":)" + json(request_id) + R"(,"method":)" + + json(constants::method) + R"(,"params":{)" + json(request_field::query) + ":" + json(sql) + members + "}}"; +} + /// Read a checked-in schema/example independently of the test's working directory. inline std::string read_document(const std::string& relative) { std::ifstream input(std::string(QUERY_PLUGIN_SOURCE_DIR) + "/" + relative); @@ -20,8 +40,9 @@ inline std::string read_document(const std::string& relative) { return {std::istreambuf_iterator(input), std::istreambuf_iterator()}; } -/// Validate the exact serialized body using the shipped Draft-04 schema. -inline void validate_document(const std::string& body, const std::string& schema_name) { +/// Check the exact serialized body against the shipped Draft-04 schema, returning the JSON pointer of +/// the first violation, or nothing when the body is valid. +inline std::optional schema_violation(const std::string& body, const std::string& schema_name) { rapidjson::Document schema_json, document; schema_json.Parse(read_document("schema/" + schema_name + ".schema.json").c_str()); BOOST_REQUIRE(!schema_json.HasParseError()); @@ -29,10 +50,17 @@ inline void validate_document(const std::string& body, const std::string& schema BOOST_REQUIRE_MESSAGE(!document.HasParseError(), body); rapidjson::SchemaDocument schema(schema_json); rapidjson::SchemaValidator validator(schema); - const bool valid = document.Accept(validator); + if (document.Accept(validator)) + return std::nullopt; rapidjson::StringBuffer path; validator.GetInvalidDocumentPointer().StringifyUriFragment(path); - BOOST_REQUIRE_MESSAGE(valid, schema_name << " at " << path.GetString() << ": " << body); + return std::string(path.GetString()); +} + +/// Validate the exact serialized body using the shipped Draft-04 schema. +inline void validate_document(const std::string& body, const std::string& schema_name) { + const auto violation = schema_violation(body, schema_name); + BOOST_REQUIRE_MESSAGE(!violation, schema_name << " at " << violation.value_or("") << ": " << body); } /// Dynamic columns impose invariants which a static JSON schema cannot express. @@ -51,6 +79,16 @@ inline void validate_response(const std::string& body) { BOOST_REQUIRE(names.insert(column["name"].as_string()).second); const auto& rows = result["rows"].get_array(); BOOST_CHECK_EQUAL(result["stats"]["returned_rows"].as_string(), std::to_string(rows.size())); + // The page window agrees with the rows and work counters it describes. + const auto& page = result[response_field::page]; + BOOST_CHECK_EQUAL(page[response_field::returned_rows].as_string(), + result[response_field::stats][response_field::returned_rows].as_string()); + const auto offset = std::stoull(page[response_field::offset].as_string()); + const auto total = std::stoull(page[response_field::total_rows].as_string()); + BOOST_CHECK_LE(rows.size(), total - std::min(offset, total)); + BOOST_CHECK_EQUAL(page[response_field::has_more].as_bool(), std::min(offset, total) + rows.size() < total); + if (!page[response_field::limit].is_null()) + BOOST_CHECK_LE(rows.size(), std::stoull(page[response_field::limit].as_string())); for (const auto& row : rows) { BOOST_REQUIRE_EQUAL(row.get_object().size(), names.size()); for (const auto& column : result["columns"].get_array()) { diff --git a/plugins/query_engine_plugin/test/test_query_execute.cpp b/plugins/query_engine_plugin/test/test_query_execute.cpp index 1b44701428..f5aaa82ced 100644 --- a/plugins/query_engine_plugin/test/test_query_execute.cpp +++ b/plugins/query_engine_plugin/test/test_query_execute.cpp @@ -63,6 +63,128 @@ BOOST_AUTO_TEST_CASE(limit_offset_and_aggregate_semantics) { execute(engine, reads, ordered_sql, query_options{.limit = std::numeric_limits::max()}).rows.size(), 3); } +/// `page` describes the window over the complete ordered output for detail and grouped queries. +BOOST_AUTO_TEST_CASE(page_window_metadata) { + read_queue reads; + query_engine engine({}, reads.create_api(*validating_node)); + const auto expect_page = [](const query_result& result, const std::string& offset, + const std::optional& limit, const std::string& returned, + const std::string& total, bool has_more) { + BOOST_CHECK_EQUAL(result.page[response_field::offset].as_string(), offset); + if (limit) + BOOST_CHECK_EQUAL(result.page[response_field::limit].as_string(), *limit); + else + BOOST_CHECK(result.page[response_field::limit].is_null()); + BOOST_CHECK_EQUAL(result.page[response_field::returned_rows].as_string(), returned); + BOOST_CHECK_EQUAL(result.page[response_field::total_rows].as_string(), total); + BOOST_CHECK_EQUAL(result.page[response_field::has_more].as_bool(), has_more); + BOOST_CHECK_EQUAL(result.page[response_field::returned_rows].as_string(), + result.stats[response_field::returned_rows].as_string()); + BOOST_CHECK_EQUAL(result.page[response_field::returned_rows].as_string(), std::to_string(result.rows.size())); + }; + expect_page(execute(engine, reads, ordered_sql), "0", std::nullopt, "3", "3", false); + expect_page(execute(engine, reads, ordered_sql, query_options{.limit = 1, .offset = 1}), "1", "1", "1", "3", true); + expect_page(execute(engine, reads, ordered_sql, query_options{.limit = 2, .offset = 1}), "1", "2", "2", "3", false); + // An offset beyond the end yields an empty, final page that still reports the requested offset. + expect_page(execute(engine, reads, ordered_sql, query_options{.offset = 5}), "5", std::nullopt, "0", "3", false); + // Likewise past the last group: the total still counts every group passing HAVING. + expect_page(execute(engine, reads, grouped_sql, query_options{.offset = 5}), "5", std::nullopt, "0", "2", false); + // An offset exactly at the end is an empty, final page too; one row earlier holds the last row. + expect_page(execute(engine, reads, ordered_sql, query_options{.offset = 3}), "3", std::nullopt, "0", "3", false); + expect_page(execute(engine, reads, ordered_sql, query_options{.offset = 2}), "2", std::nullopt, "1", "3", false); + // An output with no rows has no further page. + expect_page(execute(engine, reads, "SELECT key.id AS id FROM sample.positions WHERE amount > 1000000"), "0", + std::nullopt, "0", "0", false); + // A per-call limit of 0 returns the columns and the complete row count without any rows. + const auto columns_only = execute(engine, reads, ordered_sql, query_options{.limit = 0}); + BOOST_CHECK_EQUAL(columns_only.columns.size(), 2); + expect_page(columns_only, "0", "0", "0", "3", true); + // SQL LIMIT 0 is the same empty window over an output that still has rows. + expect_page(execute(engine, reads, std::string(ordered_sql) + " LIMIT 0"), "0", "0", "0", "3", true); + // The effective limit is the smaller of the SQL LIMIT and the per-call limit. + const auto sql_limit_one = std::string(ordered_sql) + " LIMIT 1"; + expect_page(execute(engine, reads, sql_limit_one, query_options{.limit = 2}), "0", "1", "1", "3", true); + expect_page(execute(engine, reads, sql_limit_one), "0", "1", "1", "3", true); + expect_page(execute(engine, reads, std::string(ordered_sql) + " LIMIT 5", query_options{.limit = 2}), "0", "2", "2", + "3", true); + // Detail totals count rows passing WHERE; grouped totals count groups passing HAVING. + expect_page(execute(engine, reads, "SELECT key.id AS id FROM sample.positions WHERE amount >= 60"), "0", + std::nullopt, "2", "2", false); + expect_page(execute(engine, reads, grouped_sql, query_options{.limit = 1}), "0", "1", "1", "2", true); + expect_page(execute(engine, reads, + "SELECT beneficiary, SUM(amount) AS total FROM sample.positions GROUP BY beneficiary " + "HAVING SUM(amount) > 100"), + "0", std::nullopt, "1", "1", false); + // Paging through the output one row at a time reproduces the unpaged order. + const auto all = execute(engine, reads, ordered_sql); + for (uint64_t offset = 0; offset < all.rows.size(); ++offset) { + const auto page = execute(engine, reads, ordered_sql, query_options{.limit = 1, .offset = offset}); + BOOST_REQUIRE_EQUAL(page.rows.size(), 1); + BOOST_CHECK_EQUAL(page.rows[0]["id"].as_string(), all.rows[offset]["id"].as_string()); + } + engine.stop(); +} + +/// Request options replace limit/offset and may move the deadline earlier, never later. +BOOST_AUTO_TEST_CASE(request_options_only_lower_the_deadline) { + // Time already elapsed since the request started when its options are applied. + constexpr auto elapsed_before_options = std::chrono::milliseconds(100); + // A request timeout below the configured one, so it moves the deadline earlier. + constexpr auto lowered_timeout = std::chrono::milliseconds(250); + // A request timeout above an already lowered one, which must not move the deadline later. + constexpr auto longer_timeout = std::chrono::milliseconds(500); + // A finite request timeout applied to a budget that started without a deadline. + constexpr auto unbounded_request_timeout = std::chrono::milliseconds(100); + /// A timeout below zero, which is rejected as invalid. + constexpr auto negative_timeout = std::chrono::milliseconds(-1); + const auto origin = query_budget::clock::time_point{} + std::chrono::hours(1); + auto current = origin; + const auto now = [&] { return current; }; + query_config config; + query_budget budget(config, now); + BOOST_CHECK(budget.deadline == origin + config.timeout); + budget.apply_request_options(query_options{.limit = 4, .offset = 2}); + BOOST_CHECK_EQUAL(budget.options.limit.value_or(0), 4); + BOOST_CHECK_EQUAL(budget.options.offset, 2); + BOOST_CHECK(budget.deadline == origin + config.timeout); + budget.apply_request_options(query_options{.timeout = config.timeout * 2}); + BOOST_CHECK(budget.deadline == origin + config.timeout); + BOOST_CHECK(budget.options.timeout.value_or(config.timeout) == config.timeout); + BOOST_CHECK(!budget.options.limit); + BOOST_CHECK_EQUAL(budget.options.offset, 0); + // Measured from the request's start, not from the moment the options are applied. + current = origin + elapsed_before_options; + // A timeout equal to the configured one does not lower the deadline. + budget.apply_request_options(query_options{.timeout = config.timeout}); + BOOST_CHECK(budget.deadline == origin + config.timeout); + BOOST_CHECK(budget.options.timeout.value_or(config.timeout) == config.timeout); + budget.apply_request_options(query_options{.timeout = lowered_timeout}); + BOOST_CHECK(budget.deadline == origin + lowered_timeout); + BOOST_CHECK(budget.options.timeout == lowered_timeout); + budget.apply_request_options(query_options{.timeout = lowered_timeout}); + BOOST_CHECK(budget.deadline == origin + lowered_timeout); + budget.apply_request_options(query_options{.timeout = longer_timeout}); + BOOST_CHECK(budget.deadline == origin + lowered_timeout); + BOOST_CHECK(budget.options.timeout == lowered_timeout); + current = origin + lowered_timeout; + BOOST_CHECK_EXCEPTION(budget.check(), query_error, + [](const auto& error) { return error.kind == error_kind::QUERY_TIMEOUT; }); + // An explicit no-deadline budget still accepts a finite request deadline, and no_deadline itself + // leaves it unbounded without overflowing the clock. + query_budget unbounded(config, now, query_options{.timeout = constants::no_deadline}); + unbounded.apply_request_options(query_options{.timeout = constants::no_deadline}); + BOOST_CHECK(unbounded.deadline == query_budget::clock::time_point::max()); + unbounded.apply_request_options(query_options{.timeout = unbounded_request_timeout}); + BOOST_CHECK(unbounded.deadline == unbounded.started + unbounded_request_timeout); + BOOST_CHECK(unbounded.options.timeout == unbounded_request_timeout); + // A negative timeout exceeds no limit, so none is named, and the rejected request changes nothing. + BOOST_CHECK_EXCEPTION(unbounded.apply_request_options(query_options{.timeout = negative_timeout, .limit = 1}), + query_error, + [](const auto& error) { return error.kind == error_kind::INVALID_PARAMS && !error.limit; }); + BOOST_CHECK(unbounded.deadline == unbounded.started + unbounded_request_timeout); + BOOST_CHECK(!unbounded.options.limit); +} + /// Absent and default-constructed options apply the configured deadline; only no_deadline opts out. BOOST_AUTO_TEST_CASE(default_timeout_is_configured_and_no_deadline_is_explicit) { for (const auto& options : {std::optional{}, std::optional{query_options{}}, @@ -91,9 +213,16 @@ BOOST_AUTO_TEST_CASE(option_errors_and_resource_limits) { query_config config; config.max_result_rows = 1; query_engine engine(config, reads.create_api(*validating_node)); - for (const auto timeout : {std::chrono::milliseconds(-2), constants::no_deadline - std::chrono::milliseconds(1)}) - BOOST_CHECK_EXCEPTION(execute(engine, reads, ordered_sql, query_options{.timeout = timeout}), query_error, - [](const auto& error) { return error.kind == error_kind::INVALID_PARAMS; }); + // A negative timeout names no limit; one beyond the clock's range names the timeout option. + for (const auto& [timeout, limit] : std::vector>>{ + {std::chrono::milliseconds(-2), std::nullopt }, + {constants::no_deadline - std::chrono::milliseconds(1), option::timeout_ms} + }) { + const auto expected = limit; + BOOST_CHECK_EXCEPTION( + execute(engine, reads, ordered_sql, query_options{.timeout = timeout}), query_error, + [&](const auto& error) { return error.kind == error_kind::INVALID_PARAMS && error.limit == expected; }); + } BOOST_CHECK_EXCEPTION(execute(engine, reads, ordered_sql, query_options{.timeout = std::chrono::milliseconds(0)}), query_error, [](const auto& error) { return error.kind == error_kind::QUERY_TIMEOUT; }); BOOST_CHECK_EXCEPTION(execute(engine, reads, ordered_sql), query_error, [](const auto& error) { diff --git a/plugins/query_engine_plugin/test/test_query_http.cpp b/plugins/query_engine_plugin/test/test_query_http.cpp index 67e1206660..ba6f175d86 100644 --- a/plugins/query_engine_plugin/test/test_query_http.cpp +++ b/plugins/query_engine_plugin/test/test_query_http.cpp @@ -10,6 +10,8 @@ #include #include +#include + #include #include #include @@ -496,6 +498,154 @@ BOOST_AUTO_TEST_CASE(query_http_irreversible_read_mode) { BOOST_CHECK_LE(result["state"]["block_num"].as_uint64(), result["state"]["last_irreversible_block_num"].as_uint64()); } +/// HTTP pages over the complete ordered output: pages concatenate to the full result even when that +/// result exceeds `query-max-result-rows`, which a single unpaged request still fails with QUERY_LIMIT. +BOOST_AUTO_TEST_CASE(query_http_paging_options) { + constexpr uint32_t seeded_rows = 9; + constexpr uint64_t fixture_rows = 3; + constexpr uint64_t total_rows = fixture_rows + seeded_rows; + constexpr uint64_t result_row_cap = 5; + constexpr uint64_t page_size = 4; + constexpr auto rows_sql = "SELECT key.id AS id, amount FROM sample.positions"; + chain_fixture fixture; + fixture.seed(account, fixture_rows + 1, seeded_rows, "carol"_n, 5); + fixture.produce_block(); + http_application server; + BOOST_REQUIRE( + server.initialize(true, default_read_threads, {flag(option::max_result_rows), std::to_string(result_row_cap)}) == + chain::exit_code::SUCCESS); + server.start(); + server.sync(fixture); + const auto query = [&](const std::string& members) { + const auto response = server.request(request_body(rows_sql, members)); + BOOST_REQUIRE(response.result() == http::status::ok); + return fc::json::from_string(response.body()); + }; + + const auto unpaged = query(""); + BOOST_REQUIRE(unpaged.get_object().contains(response_field::error)); + BOOST_CHECK_EQUAL(unpaged[response_field::error][response_field::data][response_field::kind].as_string(), + magic_enum::enum_name(error_kind::QUERY_LIMIT)); + BOOST_CHECK_EQUAL(unpaged[response_field::error][response_field::data][response_field::limit].as_string(), + option::max_result_rows); + + // An explicit limit is still capped: one row above the cap fails the same way, the cap itself pages. + const auto above_cap = query(R"(,"limit":)" + std::to_string(result_row_cap + 1)); + BOOST_REQUIRE(above_cap.get_object().contains(response_field::error)); + const auto& above_cap_data = above_cap[response_field::error][response_field::data]; + BOOST_CHECK_EQUAL(above_cap_data[response_field::kind].as_string(), magic_enum::enum_name(error_kind::QUERY_LIMIT)); + BOOST_CHECK_EQUAL(above_cap_data[response_field::limit].as_string(), option::max_result_rows); + const auto at_cap = query(R"(,"limit":)" + std::to_string(result_row_cap)); + BOOST_REQUIRE_MESSAGE(at_cap.get_object().contains(response_field::result), + fc::json::to_string(at_cap, fc::time_point::maximum())); + const auto& at_cap_page = at_cap[response_field::result][response_field::page]; + BOOST_CHECK_EQUAL(at_cap[response_field::result][response_field::rows].get_array().size(), result_row_cap); + BOOST_CHECK_EQUAL(at_cap_page[response_field::limit].as_string(), std::to_string(result_row_cap)); + BOOST_CHECK_EQUAL(at_cap_page[response_field::total_rows].as_string(), std::to_string(total_rows)); + BOOST_CHECK(at_cap_page[response_field::has_more].as_bool()); + + std::vector ids; + for (uint64_t offset = 0;; offset += page_size) { + BOOST_REQUIRE_LT(offset, total_rows); + const auto response = + query(R"(,"limit":)" + std::to_string(page_size) + R"(,"offset":)" + std::to_string(offset)); + BOOST_REQUIRE_MESSAGE(response.get_object().contains(response_field::result), + fc::json::to_string(response, fc::time_point::maximum())); + const auto& result = response[response_field::result]; + BOOST_CHECK_EQUAL(result[response_field::schema_version].as_string(), constants::schema_version); + BOOST_CHECK(result[response_field::complete].as_bool()); + BOOST_CHECK_EQUAL(result[response_field::page][response_field::offset].as_string(), std::to_string(offset)); + BOOST_CHECK_EQUAL(result[response_field::page][response_field::limit].as_string(), std::to_string(page_size)); + BOOST_CHECK_EQUAL(result[response_field::page][response_field::total_rows].as_string(), + std::to_string(total_rows)); + for (const auto& row : result[response_field::rows].get_array()) + ids.push_back(row["id"].as_string()); + if (!result[response_field::page][response_field::has_more].as_bool()) + break; + } + BOOST_REQUIRE_EQUAL(ids.size(), total_rows); + for (uint64_t i = 0; i < total_rows; ++i) + BOOST_CHECK_EQUAL(ids[i], std::to_string(i + 1)); + + const auto beyond = query(R"(,"offset":)" + std::to_string(total_rows + fixture_rows)); + BOOST_REQUIRE(beyond.get_object().contains(response_field::result)); + BOOST_CHECK(beyond[response_field::result][response_field::rows].get_array().empty()); + BOOST_CHECK(!beyond[response_field::result][response_field::page][response_field::has_more].as_bool()); + BOOST_CHECK_EQUAL(beyond[response_field::result][response_field::page][response_field::total_rows].as_string(), + std::to_string(total_rows)); + + const auto columns_only = query(R"(,"limit":0)"); + BOOST_REQUIRE(columns_only.get_object().contains(response_field::result)); + BOOST_CHECK(columns_only[response_field::result][response_field::rows].get_array().empty()); + BOOST_CHECK_EQUAL(columns_only[response_field::result][response_field::columns].get_array().size(), 2); + BOOST_CHECK_EQUAL(columns_only[response_field::result][response_field::page][response_field::limit].as_string(), + "0"); + BOOST_CHECK(columns_only[response_field::result][response_field::page][response_field::has_more].as_bool()); + + // A lowered timeout_ms the query fits within still returns its page. + const auto lowered_timeout_ms = std::chrono::milliseconds(default_query_timeout).count() - 1; + const auto lowered = query(R"(,"limit":1,"timeout_ms":)" + std::to_string(lowered_timeout_ms)); + BOOST_REQUIRE_MESSAGE(lowered.get_object().contains(response_field::result), + fc::json::to_string(lowered, fc::time_point::maximum())); + BOOST_CHECK_EQUAL(lowered[response_field::result][response_field::rows].get_array().size(), 1); + + // timeout_ms may only lower the configured deadline; malformed members are invocation errors. + const auto raised = query(R"(,"limit":1,"timeout_ms":)" + + std::to_string(std::chrono::milliseconds(default_query_timeout).count() + 1)); + BOOST_CHECK_EQUAL(raised[response_field::error][response_field::data][response_field::kind].as_string(), + magic_enum::enum_name(error_kind::INVALID_PARAMS)); + BOOST_CHECK_EQUAL(raised[response_field::error][response_field::data][response_field::limit].as_string(), + option::timeout_ms); + for (const auto* members : {R"(,"limit":"1")", R"(,"offset":-1)", R"(,"limit":1.5)", R"(,"cursor":1)"}) + BOOST_CHECK_EQUAL(query(members)[response_field::error][response_field::data][response_field::kind].as_string(), + magic_enum::enum_name(error_kind::INVALID_PARAMS)); +} + +/// A request's timeout_ms lowers its deadline, measured from ingress: while the application thread +/// holds the write window no read can capture, so the lowered request times out while the same query +/// under the configured deadline completes once the window opens. +BOOST_AUTO_TEST_CASE(query_http_timeout_ms_lowers_the_deadline) { + constexpr auto lowered_timeout = std::chrono::milliseconds(200); + chain_fixture fixture; + http_application server; + BOOST_REQUIRE(server.initialize(true) == chain::exit_code::SUCCESS); + server.start(); + server.sync(fixture); + auto entered = std::make_shared>(); + auto reached = entered->get_future(); + auto resume = std::make_shared>(); + const auto release = resume->get_future().share(); + auto blocked = server.post_write([entered, release] { + entered->set_value(); + if (release.wait_for(test_wait) != std::future_status::ready) + throw std::runtime_error("Deadline barrier was not released"); + }); + BOOST_REQUIRE(reached.wait_for(test_wait) == std::future_status::ready); + auto lowered = std::async(std::launch::async, [&] { + return server.request(request_body(selected_sql, R"(,"timeout_ms":)" + std::to_string(lowered_timeout.count()))); + }); + auto configured = std::async(std::launch::async, [&] { return server.request(request_body(selected_sql)); }); + const auto lowered_ready = lowered.wait_for(test_wait); + // With the window still closed, the configured request outlives the lowered deadline by at least + // another lowered interval rather than merely not having finished at the same instant. + const auto configured_pending = configured.wait_for(lowered_timeout); + resume->set_value(); + BOOST_REQUIRE(lowered_ready == std::future_status::ready); + BOOST_CHECK(configured_pending == std::future_status::timeout); + const auto timed_out = fc::json::from_string(lowered.get().body()); + BOOST_REQUIRE_MESSAGE(timed_out.get_object().contains(response_field::error), + fc::json::to_string(timed_out, fc::time_point::maximum())); + BOOST_CHECK_EQUAL(timed_out[response_field::error][response_field::data][response_field::kind].as_string(), + magic_enum::enum_name(error_kind::QUERY_TIMEOUT)); + BOOST_REQUIRE(blocked.wait_for(test_wait) == std::future_status::ready); + blocked.get(); + BOOST_REQUIRE(configured.wait_for(test_wait) == std::future_status::ready); + const auto completed = fc::json::from_string(configured.get().body()); + BOOST_REQUIRE_MESSAGE(completed.get_object().contains(response_field::result), + fc::json::to_string(completed, fc::time_point::maximum())); + BOOST_CHECK_EQUAL(completed[response_field::result][response_field::rows].get_array().size(), 1); +} + /// Saturating admission rejects another HTTP request without waiting for chain work. BOOST_AUTO_TEST_CASE(query_http_admission_while_chain_queue_is_blocked) { chain_fixture fixture; diff --git a/plugins/query_engine_plugin/test/test_query_rpc.cpp b/plugins/query_engine_plugin/test/test_query_rpc.cpp index 921102f50a..0484773d52 100644 --- a/plugins/query_engine_plugin/test/test_query_rpc.cpp +++ b/plugins/query_engine_plugin/test/test_query_rpc.cpp @@ -33,6 +33,24 @@ query_request parse_id_request(const std::string& id) { ",\"method\":\"query.execute\",\"params\":{\"query\":\"SELECT * FROM sample.wide\"}}", budget); } +/// Parse a request whose params carry `query` plus the given members. +query_request parse_params_request(const std::string& members, const query_config& config = {}) { + query_budget budget(config); + return parse_request(request_body(wide_select_sql, members), budget); +} +/// A rejected params member is an invocation error: INVALID_PARAMS on the wire with the echoed ID. +fc::variant expect_invalid_params(const std::string& members, const query_config& config = {}) { + const auto request = parse_params_request(members, config); + BOOST_REQUIRE_MESSAGE(request.invocation_error, members); + BOOST_CHECK_MESSAGE(request.invocation_error->kind == error_kind::INVALID_PARAMS, members); + const auto envelope = create_error(&request, *request.invocation_error); + validate_document(fc::json::to_string(envelope, fc::time_point::maximum()), "query-response"); + BOOST_CHECK_EQUAL(envelope["id"].as_string(), request_id); + BOOST_CHECK_EQUAL(envelope[response_field::error][response_field::code].as_int64(), -32602); + BOOST_CHECK_EQUAL(envelope[response_field::error][response_field::data][response_field::kind].as_string(), + magic_enum::enum_name(error_kind::INVALID_PARAMS)); + return envelope; +} } // namespace BOOST_AUTO_TEST_SUITE(query_rpc) @@ -76,6 +94,107 @@ BOOST_AUTO_TEST_CASE(envelopes_notifications_and_invocation_errors) { BOOST_CHECK(invalid.invocation_error->kind == error_kind::INVALID_PARAMS); } +/// `{query}` alone keeps every option at its default; limit, offset and timeout_ms are exact JSON +/// integers, including fraction and exponent spellings of an integral value. +BOOST_AUTO_TEST_CASE(paging_parameters_parse_exactly) { + const auto plain = parse_params_request(""); + BOOST_REQUIRE(!plain.invocation_error); + BOOST_CHECK_EQUAL(plain.query, wide_select_sql); + BOOST_CHECK(!plain.options.limit); + BOOST_CHECK_EQUAL(plain.options.offset, 0); + BOOST_CHECK(!plain.options.timeout); + for (const auto& [members, limit] : std::vector>{ + {R"(,"limit":2)", 2 }, + {R"(,"limit":2e0)", 2 }, + {R"(,"limit":20e-1)", 2 }, + {R"(,"limit":2.0)", 2 }, + {R"(,"limit":0)", 0 }, + {R"(,"limit":9007199254740991)", constants::max_request_integer} + }) { + const auto request = parse_params_request(members); + BOOST_REQUIRE_MESSAGE(!request.invocation_error, members); + BOOST_REQUIRE(request.options.limit); + BOOST_CHECK_EQUAL(*request.options.limit, limit); + } + for (const auto& [members, offset] : std::vector>{ + {R"(,"offset":3)", 3 }, + {R"(,"offset":-0)", 0 }, + {R"(,"offset":3E+0)", 3 }, + {R"(,"offset":9007199254740991)", constants::max_request_integer} + }) { + const auto request = parse_params_request(members); + BOOST_REQUIRE_MESSAGE(!request.invocation_error, members); + BOOST_CHECK_EQUAL(request.options.offset, offset); + } + const auto timed = parse_params_request(R"(,"timeout_ms":5e2,"offset":1,"limit":4)"); + BOOST_REQUIRE(!timed.invocation_error); + BOOST_CHECK(timed.options.timeout == std::chrono::milliseconds(500)); + BOOST_CHECK_EQUAL(timed.options.offset, 1); + BOOST_CHECK_EQUAL(timed.options.limit.value_or(0), 4); + // The configured timeout itself is the largest accepted value. + const auto configured = parse_params_request(R"(,"timeout_ms":)" + std::to_string(defaults::timeout.count())); + BOOST_REQUIRE(!configured.invocation_error); + BOOST_CHECK(configured.options.timeout == defaults::timeout); + constexpr auto raised_timeout_factor = 5; + constexpr auto above_default_timeout = defaults::timeout + std::chrono::milliseconds(1); + query_config longer; + longer.timeout = defaults::timeout * raised_timeout_factor; + const auto raised = + parse_params_request(R"(,"timeout_ms":)" + std::to_string(above_default_timeout.count()), longer); + BOOST_REQUIRE(!raised.invocation_error); + BOOST_CHECK(raised.options.timeout == above_default_timeout); +} + +/// Every malformed, out-of-range, unknown or duplicated params member is INVALID_PARAMS. +BOOST_AUTO_TEST_CASE(paging_parameters_reject_invalid_members) { + for (const auto* members : {R"(,"limit":"2")", + R"(,"limit":-1)", + R"(,"limit":1.5)", + R"(,"limit":9007199254740992)", + R"(,"limit":1e16)", + R"(,"limit":null)", + R"(,"limit":true)", + R"(,"limit":[2])", + R"(,"limit":{"value":2})", + R"(,"offset":"0")", + R"(,"offset":-1)", + R"(,"offset":0.5)", + R"(,"offset":9007199254740992)", + R"(,"offset":null)", + R"(,"timeout_ms":0)", + R"(,"timeout_ms":-5)", + R"(,"timeout_ms":"500")", + R"(,"timeout_ms":1.5)", + R"(,"timeout_ms":null)", + R"(,"owner":"sample")", + R"(,"limit":1,"limit":2)", + R"(,"query":"SELECT * FROM sample.wide")"}) { + const auto envelope = expect_invalid_params(members); + BOOST_CHECK_MESSAGE(envelope[response_field::error][response_field::data][response_field::limit].is_null(), + members); + } + BOOST_CHECK_EQUAL( + expect_invalid_params(R"(,"limit":-1)")[response_field::error][response_field::message].as_string(), + "limit must be an integer in [0, 9007199254740991]"); + BOOST_CHECK_EQUAL( + expect_invalid_params(R"(,"timeout_ms":0)")[response_field::error][response_field::message].as_string(), + "timeout_ms must be an integer in [1, 9007199254740991]"); + // A timeout above the configured deadline names the option it may only lower. + for (const auto& timeout : + {std::to_string(defaults::timeout.count() + 1), std::to_string(constants::max_request_integer)}) { + const auto envelope = expect_invalid_params(R"(,"timeout_ms":)" + timeout); + BOOST_CHECK_EQUAL(envelope[response_field::error][response_field::data][response_field::limit].as_string(), + option::timeout_ms); + } + for (const auto* params : {R"([])", R"("SELECT")", R"(null)", R"({})"}) { + query_budget budget({}); + const auto request = parse_request( + std::string(R"({"jsonrpc":"2.0","id":7,"method":"query.execute","params":)") + params + "}", budget); + BOOST_REQUIRE_MESSAGE(request.invocation_error, params); + BOOST_CHECK(request.invocation_error->kind == error_kind::INVALID_PARAMS); + } +} + BOOST_AUTO_TEST_CASE(startup_rejects_invalid_unsigned_options) { for (const auto* value : {"-1", "0", "4294967296", "1.5", "nonsense"}) BOOST_CHECK_THROW(parse_arguments({std::string("--") + option::worker_threads, value}), query_error); diff --git a/plugins/query_engine_plugin/test/test_query_schema.cpp b/plugins/query_engine_plugin/test/test_query_schema.cpp index fade7bfe41..acad73731a 100644 --- a/plugins/query_engine_plugin/test/test_query_schema.cpp +++ b/plugins/query_engine_plugin/test/test_query_schema.cpp @@ -1,9 +1,73 @@ #include "query_schema.hpp" +#include + +namespace { +using namespace sysio::query_engine; +using namespace sysio::query_engine::test; + +constexpr auto request_schema = "query-request"; +constexpr auto response_schema = "query-response"; + +/// The paged response example with its `result` changed by `mutate`. +std::string paged_response(const std::function& mutate) { + const auto document = fc::json::from_string(read_document("examples/response-paged.json")); + fc::mutable_variant_object result(document[response_field::result].get_object()); + mutate(result); + fc::mutable_variant_object envelope(document.get_object()); + envelope.set(response_field::result, fc::variant_object(result)); + return fc::json::to_string(fc::variant_object(envelope), fc::time_point::maximum()); +} + +/// The paged response example with one `page` member replaced (or removed when `value` is absent). +std::string paged_response_with(const std::string& member, const std::optional& value) { + return paged_response([&](fc::mutable_variant_object& result) { + fc::mutable_variant_object page(result[response_field::page].get_object()); + if (value) + page.set(member, *value); + else + page.erase(member); + result.set(response_field::page, fc::variant_object(page)); + }); +} +} // namespace + /// Documentation examples are subject to the same wire validation as runtime responses. BOOST_AUTO_TEST_CASE(query_documentation_examples) { - using namespace sysio::query_engine::test; - validate_document(read_document("examples/request.json"), "query-request"); + validate_document(read_document("examples/request.json"), request_schema); + validate_document(read_document("examples/request-paged.json"), request_schema); validate_response(read_document("examples/response.json")); + validate_response(read_document("examples/response-paged.json")); validate_response(read_document("examples/error.json")); } + +/// The request schema admits the paging members only as bounded JSON integers. +BOOST_AUTO_TEST_CASE(query_request_schema_paging_members) { + for (const auto* members : + {R"(,"limit":0)", R"(,"limit":9007199254740991)", R"(,"offset":0)", R"(,"offset":9007199254740991)", + R"(,"timeout_ms":1)", R"(,"limit":2,"offset":2,"timeout_ms":500)"}) + BOOST_CHECK_MESSAGE(!schema_violation(request_body(wide_select_sql, members), request_schema), members); + for (const auto* members : {R"(,"limit":-1)", R"(,"limit":1.5)", R"(,"limit":"2")", R"(,"limit":null)", + R"(,"limit":9007199254740992)", R"(,"offset":-1)", R"(,"offset":true)", + R"(,"timeout_ms":0)", R"(,"timeout_ms":9007199254740992)", R"(,"owner":"sample")"}) + BOOST_CHECK_MESSAGE(schema_violation(request_body(wide_select_sql, members), request_schema), members); +} + +/// The response schema requires the complete, strictly shaped page object. +BOOST_AUTO_TEST_CASE(query_response_schema_page_member) { + BOOST_CHECK(!schema_violation(paged_response_with(response_field::limit, fc::variant()), response_schema)); + for (const auto* member : {response_field::offset, response_field::limit, response_field::returned_rows, + response_field::total_rows, response_field::has_more}) + BOOST_CHECK_MESSAGE(schema_violation(paged_response_with(member, std::nullopt), response_schema), member); + BOOST_CHECK(schema_violation(paged_response_with(response_field::has_more, fc::variant("true")), response_schema)); + BOOST_CHECK( + schema_violation(paged_response_with(response_field::total_rows, fc::variant(uint64_t{7})), response_schema)); + BOOST_CHECK(schema_violation(paged_response_with(response_field::offset, fc::variant("-1")), response_schema)); + BOOST_CHECK( + schema_violation(paged_response_with(response_field::returned_rows, fc::variant(uint64_t{2})), response_schema)); + BOOST_CHECK(schema_violation(paged_response_with(response_field::limit, fc::variant("02")), response_schema)); + BOOST_CHECK(schema_violation(paged_response_with(response_field::limit, fc::variant(uint64_t{2})), response_schema)); + BOOST_CHECK(schema_violation(paged_response_with("cursor", fc::variant("next")), response_schema)); + BOOST_CHECK(schema_violation( + paged_response([](fc::mutable_variant_object& result) { result.erase(response_field::page); }), response_schema)); +}