Repository navigation
test: differential parity run against Loki and a ranked gap inventory - #657
Merged
Merged
Conversation
bench/parity: a differential runner for the proxy against Loki on an isolated stack with identical data. It builds a corpus from Loki v3.7.7's own LogQL test suites (selectors moved onto the seeded data), the repository's parity cases, a sweep generated from what Loki reports for the window (labels, values, detected fields and the stages Explore and Logs Drilldown build on them, the metadata endpoints, patterns, format_query, tail) and the Grafana Explore and Logs Drilldown requests captured by bench/visual, replayed through Grafana. It proves both sides healthy and equal before and after (per-slice line counts, Loki's blank window, known and unchanged Loki and VictoriaLogs restart counts, no VictoriaLogs failure in the proxy log), diffs every answer semantically with and without categorize-labels, reconfirms samples of every difference, and clusters the rest by endpoint, query shape and kind of difference. --keep-bodies and --rediff let a later diff rule be applied to a recorded run offline. The run on main at fcaccfc is recorded in conformance/registry/generated/parity-discovery.json; every cluster is claimed by exactly one registry case with a gap block (Loki reference, proxy behaviour, impact, effort, code area, planned test), recorded deviations by the case id the diff names. conformance/scripts/parity_gaps.py ranks them into conformance/reports/parity-gaps.md and fails the gate on an unclaimed or doubly claimed cluster, a deviation that names no recorded case, or `status: fixed` without `fixed_by`. bench/visual/stack.py takes --overlay for a further compose file; the parity overlay raises Loki's gRPC message limits so per-stream metric answers are not dropped between Loki's querier and frontend.
github-actions
Bot
added
size/XL
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| a = ap.parse_args() | ||
| rows = load_rows(os.path.join(a.run_dir, "results.jsonl")) | ||
| health_path = os.path.join(a.run_dir, "health.json") | ||
| health = json.load(open(health_path)) if os.path.exists(health_path) else {} |
| for raw in sorted(extract_strings(glob.glob(os.path.join(ROOT, "test", "e2e-compat", "*_test.go")))): | ||
| add("repo", raw, "test/e2e-compat") | ||
| matrix = os.path.join(ROOT, "test", "e2e-compat", "query-semantics-matrix.json") | ||
| for case in json.load(open(matrix))["cases"]: |
| matrix = os.path.join(ROOT, "test", "e2e-compat", "query-semantics-matrix.json") | ||
| for case in json.load(open(matrix))["cases"]: | ||
| add("repo", case["query"], f"query-semantics-matrix:{case['id']}") | ||
| shapes = json.load(open(os.path.join(ROOT, "bench", "ab", "shapes.json"))) |
| if shape.get("query"): | ||
| add("repo", shape["query"], f"shapes:{name}:{shape['name']}") | ||
| for path in sorted(glob.glob(os.path.join(ROOT, "conformance", "registry", "cases", "**", "*.yaml"), recursive=True)): | ||
| text = open(path).read() |
| for directory in capture_dirs: | ||
| for path in sorted(glob.glob(os.path.join(directory, "**", "*.json"), recursive=True)): | ||
| try: | ||
| doc = json.load(open(path)) |
| ap.add_argument("--ref", required=True) | ||
| ap.add_argument("--out", default=OUT) | ||
| a = ap.parse_args() | ||
| data = json.load(open(os.path.join(a.run_dir, "clusters.json"))) |
| ap.add_argument("--out", default=OUT) | ||
| a = ap.parse_args() | ||
| data = json.load(open(os.path.join(a.run_dir, "clusters.json"))) | ||
| health = json.load(open(os.path.join(a.run_dir, "health.json"))) |
| # Runs merged into this one (an earlier pass, a rerun of one source) keep their own proof. | ||
| passes = {} | ||
| for path in sorted(glob.glob(os.path.join(a.run_dir, "health-*.json"))): | ||
| h = json.load(open(path)) |
| "fallback_after_4xx": 0, "partial_after_5xx": 0, "lines": 0} | ||
| if not path or not os.path.exists(path): | ||
| return {} | ||
| for line in open(path, errors="ignore"): |
| help="confirm the differences of an existing RUN_DIR/results.jsonl instead of running the corpus") | ||
| a = ap.parse_args() | ||
| if a.state: | ||
| state = json.load(open(a.state)) |
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokePerformance smoke was skipped for this PR because no perf-sensitive paths changed. State
|
Contributor
✅ Visual smoke: passed with warningsBase
Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-05T21%3A27%3
dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-05T21%3A27%3
Core set montages (3): base | PR | Lokidd-landing-1h
explore-logs-plain-1h
explore-metric-plain-1h
All montages of this run: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A reusable differential runner (
bench/parity) that compares the proxy with Loki v3.7.7 on identical data, and a registry-backed, ranked list of every known difference (conformance/reports/parity-gaps.md). Discovery and registration only: no proxy code change.bench/parity/corpus.py: corpus from Loki v3.7.7's ownpkg/logqltest suites (selectors moved onto the seeded data, sharding/template fixtures dropped, ranges clamped to the seeded hour), the repo's parity cases (test/e2e-compat,query-semantics-matrix.json,bench/ab/shapes.json, registry case requests), a seeded sweep generated from Loki's own answers (labels/values, every service's detected fields with the stages Explore and Drilldown build, index/stats, volume, volume_range, detected_*, patterns, format_query, tail), and the Grafana Explore + Drilldown requests captured bybench/visual(replayed through Grafana against both datasources).bench/parity/run.py: proves both sides healthy before and after (equal line counts per 10-minute slice, Loki out of its blank window, Loki and VictoriaLogs restart counts known and unchanged, no VictoriaLogs 5xx / fallback-after-5xx in the proxy log), runs query endpoints with and withoutcategorize-labels, diffs semantically (differ.py: entry order vs direction, duplicate label sets, scalars as numbers), reconfirms up to three samples of every difference signature, clusters by endpoint + query shape (operation, parser, stages, line/label/pattern filter) + kind (cluster.py).--keep-bodies/--rediffre-apply diff rules to a recorded run offline.bench/parity/publish.pyrecords a run inconformance/registry/generated/parity-discovery.json;conformance/scripts/parity_gaps.pyranks the registry'sgap:cases intoconformance/reports/parity-gaps.md. The conformance gate fails on an unclaimed or doubly claimed cluster, a recorded deviation that names no documented/owner-kept/upstream case, orstatus: fixedwithoutfixed_by(a fix PR retires a gap without a rerun).bench/visual/stack.py --overlay;bench/parity/docker-compose.parity.ymlraises Loki's gRPC message limits (8 MiB in the e2e config dropped per-stream metric answers between querier and frontend).Recorded run: main at fcaccfc (before #652 rewrote the tail and log-row label conversion)
Isolated Colima stack, Loki v3.7.7, VictoriaLogs v1.52.0, proxy built from fcaccfc. #5 (non-sum aggregation) and #16 (tail history) were verified still present on c4ebcd9 (after #652); the log-row label cases carry a note to recheck on current code before fixing.
Both sides healthy: 291,425 lines on Loki and VictoriaLogs in 6/6 equal 10-minute slices, 15/15 services in Loki's range-metric path, Loki and VictoriaLogs restart counts 0 before and after every pass (full pass, repo rerun, confirmation pass). Proxy log over the repo rerun and the confirmation pass: 0 VictoriaLogs 5xx, 0 fallback/partial after a 5xx; 277 VictoriaLogs 4xx and 2 fallbacks after a 4xx (queries the proxy translated wrongly, i.e. the measured differences); the full pass's log was replaced when the proxies restarted. 12 differences did not reproduce on a second request and are blocked, not counted. 600 gap signatures + 5 recorded-deviation clusters, all claimed by exactly one of 66 registry cases (47 open, 4 owner-kept, 13 documented, 2 upstream). The entry-order and duplicate-series checks were added after the run; the run stored facets, not bodies, so they apply from the next run (
--keep-bodies); no scalar/string difference was recorded, so the numeric scalar comparison changes nothing in it.Ranked differences (top 30; full list in conformance/reports/parity-gaps.md)
semantics/extracted-suffix-collisionsemantics/unwrap-sample-validityprofiles/structured-metadata-label-collisionprofiles/keep-removes-derived-service-name| keepremoves service_name the proxy derived for a stream without onesemantics/non-sum-vector-aggregation-unreducedsemantics/parenthesized-log-rangesemantics/line-filter-or-alternativesoralternatives are rejectedseverity/detected-level-stage-side-effectssemantics/logfmt-existence-filter-ungrouped-sum| logfmt | key!=""returns no seriesparsed-identity/bare-parser-metric-duplicates-extracted| jsonmetric labels each series with every parsed key twice, once with _extractedsemantics/topk-bottomk-range-seriesprofiles/keep-structured-metadata-key| keepof a structured-metadata key drops itsemantics/label-filter-expression-syntaxprofiles/json-extraction-list-exposes-only-listed| json msg="msg",| logfmt op) expose level from the linesemantics/first-last-over-time-translationsemantics/tail-history-start-limitsemantics/binary-ignoring-matching+ ignoring(label)between two sums joins the wrong seriessemantics/parse-error-labels-on-log-linessemantics/absent-over-time-semanticssemantics/stddev-stdvar-grouped-valuessemantics/label-replace-new-labelquality/label-values-chunk-windowsemantics/pattern-line-filter|> "",!> "") selects the opposite linessemantics/log-entries-differ-same-countprofiles/label-filter-regexp-anchoringquality/label-cache-aged-labels-keptsemantics/unwrap-conversion-errorsemantics/unwrap-range-last-pointenddrilldown/detected-field-valuesprofiles/json-number-label-textVisual / A-B proof
Not applicable: no proxy code change (tooling, tests and registry only).
Gates
python3 scripts/ci/check_conformance.py: ok (parity registration: 66 cases, 605 clusters, 0 problems)python3 scripts/ci/check_changelog_pr.py --base origin/main --head HEAD: okpython3 bench/ab/selection.py --check,python3 bench/visual/plan.py --check: okpython3 -m unittest discover -s bench/parity/tests(44, wired into ci.yaml),bench/visual/tests(66),bench/ab/tests(29): ok