Repository navigation
Measure public cache effectiveness and invalidation causes - #76
Conversation
PR reader API performanceResult: INCONCLUSIVE 8 inconclusive. Zero request failures across all six runs. Base Three independent runs per revision, each on a fresh GitHub-hosted ARM64 runner. 16 users, 60 seconds/run, 1,000 synthetic articles, cache off, five PgBouncer slots. Latency comparisonAll latency values are in milliseconds; lower is better. Values are medians of three runs.
Additional latency and throughput metricsEach cell shows base → PR. Latencies are in milliseconds.
Individual runs
How to interpret this reportRegressed / Improved: a p95 regression needs >20% and >20 ms increase in the medians and at least two head samples versus the base median. Improvements use the inverse threshold. Within threshold means the change did not meet both thresholds. Inconclusive: a within-revision p95 range >35% of its median and >20 ms is too noisy for a confident latency verdict. Inconclusive latency comparisons are non-blocking: the check passes without claiming an improvement. Confirmed regressions and test/report errors fail the PR gate. Throughput is descriptive: paced users do not measure maximum capacity. This uncached public API test does not establish production health, browser performance or worker capacity. Separate runners can differ in hardware or host load; repetitions and noise checks reduce but cannot eliminate that uncertainty. Full HTML/CSV reports and logs · attempt 1 |
Code Review ✅ Approved🟡 Medium risk · Adds route and writer invalidation telemetry plus CLI runtime lifecycle changes Adds bounded route templates and invalidation cause tracking to public cache observability, enabling per-route hit ratio and traffic analysis. Production data shows 33.2% hits over 24 hours; this PR instruments why entries miss and whether broad invalidation is responsible. Writer delivery is carefully measured across long-lived services, short-lived workers, and CLI commands with proper counter lifecycle and OTLP delta exports. Comprehensive regression testing covers cache decisions, invalidation causes, commit/rollback behavior, Redis failures, and route separation. No issues found. OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
|
1 participant
What changes and why?
Production public response caching lacks route labels and invalidation measurements. A read-only Prometheus snapshot at 2026-10-04 00:52 UTC found 33.2% hits over 24 hours (about 76,108 hits / 152,842 misses) and 35.7% over one hour. Cache-unavailable bypasses were about 0.024% of 24-hour decisions. Hit ratio uses hits / (hits + misses), with bypasses tracked separately. These aggregates do not establish why entries miss or prove broad invalidation is responsible.
This PR adds bounded route templates to the final HIT/MISS/BYPASS decision and records generation invalidation attempts separately from their contributing causes. A transaction counts once, even when it touches several public tables. ORM and bulk writes report table causes after the outer commit; profile, avatar, search-index, and manual clears report explicit causes. Outcomes distinguish successful generation changes, Redis unavailability, and disabled caching. Cache keys, query strings, IDs, and supplied values never become labels.
Writer delivery matters to the measurement: RQ children reuse pod identities but reset counters, and CLI writers previously did not initialize telemetry. Short-lived writer counters now export OTLP deltas with a final shutdown flush; CLI commands that can change public data own and close their runtime. Worker/scheduler entrypoints retain their existing runtime ownership, and parent scrapes remain cumulative. The live production collector was confirmed to have stream-affine routing and
deltatocumulativeenabled.The observability guide includes route hit ratio, traffic/bypass, and writer-cause queries. Writer totals use OTLP-only production series: filtering by Kubernetes namespace would omit work-horse data, while summing scrapes and OTLP would duplicate long-lived services.
Validation
Updated against master after #75 and the separate reader touch-fixture prerequisite #87. Hosted CI passes on
4fdc5a8efbf468202758013caff5f07327370377:CI required, all CodeQL analyses, SonarQube and SonarCloud Code Analysis. Auto-merge remains disabled.Rollout notes
Cache domains, keys, TTLs, invalidation triggers, and generation fencing stay unchanged. No migration or cache clear is required. Roll out the API, user/admin API, scheduler, indexer, workers, and CLI together to measure every writer. Short-lived writer counter exports require an OTLP collector that accumulates deltas; production already has this configuration. Other environments must provide equivalent handling. Collector outages or unconfigured CLI export can still lose observations.
After deployment, verify writer coverage and delivery, then compare a full post-rollout window of route traffic, hit ratio, bypasses, successful invalidations, latency, and database pressure. Causes are overlapping transaction categories and must not be summed as invalidation counts. Individual misses cannot be attributed to TTL expiry or invalidation with these aggregate counters. Whether broad invalidation actually explains the hit rate remains open until that evidence is collected. Cache boundary changes belong in a follow-up PR supported by those measurements.
This branch is based directly on
masterand contains only this cache-observability task; earlier audit fixes remain in their separate PRs. No production deployment is included.Ready for review