diff --git a/docs/api-reference.md b/docs/api-reference.md index 9b87a2a9..c13fc546 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -838,11 +838,16 @@ HTTP server or register a `/metrics` route; wire up `prometheus_client` expositi names carry no `cachekit_` prefix: - `cache_operations_total` - Operation counter. Labels: `operation`, `namespace`, `success`, `serializer` -- `redis_cache_operations_total` - Load-control operation counter. Labels: `operation`, `status`, `serializer`, `namespace` +- `redis_cache_operations_total` - Load-control rejection counter. Labels: `operation`, `status`, `serializer`, `namespace` - `cache_operation_duration_ms` - Operation latency histogram (milliseconds). Labels: `operation`, `namespace`, `serializer` - `cache_operation_size_bytes` - Operation payload size histogram (bytes). Labels: `operation`, `namespace`, `serializer` - `circuit_breaker_state` - Circuit breaker state gauge (0=CLOSED, 1=OPEN, 2=HALF_OPEN). Labels: `namespace`, `state` +The `serializer` label is the tier that served the record, not the `@cache(serializer=...)` +preset: `rust` = L2 backend path, `l1_memory` = L1 in-memory hit; `unknown` marks a record +emitted without the label. `redis_cache_operations_total` is emitted only on backpressure +rejection (`operation="backpressure"`, `status="rejected"`, empty `serializer` and `namespace`). + See the [Prometheus Metrics guide](features/prometheus-metrics.md) for exposition setup, query examples, and alerting rules. diff --git a/docs/features/distributed-locking.md b/docs/features/distributed-locking.md index bec2efa3..04a6d085 100644 --- a/docs/features/distributed-locking.md +++ b/docs/features/distributed-locking.md @@ -318,9 +318,10 @@ async def fetch_sensitive(x): Lock waiters that time out log a `Failed to acquire lock for {key} after 5.0s` warning; lock backend errors log a `Lock operation failed … executing without -lock` warning. For miss-rate monitoring (stampede detection), use the `status` -label on `redis_cache_operations_total` — see -[Prometheus Metrics](prometheus-metrics.md). +lock` warning. For miss-rate monitoring (stampede detection), watch `operation="set"` on +`cache_operations_total` as a cache-write proxy — misses write back, but it can +double-count (when stats collection is on) or miss failed writes, so treat it as a proxy, +not an exact miss count — see [Prometheus Metrics](prometheus-metrics.md). --- @@ -335,7 +336,7 @@ A: Two things to check: 2. The backend must implement `LockableBackend` (`RedisBackend`, `CachekitIOBackend`). Check with `from cachekit.backends.base import LockableBackend; isinstance(backend, LockableBackend)`. **Q: How do I know if stampedes are happening?** -A: Check Prometheus: a spike in `rate(redis_cache_operations_total{status="miss"}[1m])` = stampede risk. See [Prometheus Metrics](prometheus-metrics.md). +A: Check Prometheus: a spike in `rate(cache_operations_total{operation="set"}[1m])` (a cache-write proxy — misses write back) suggests stampede risk. See [Prometheus Metrics](prometheus-metrics.md). --- diff --git a/docs/features/prometheus-metrics.md b/docs/features/prometheus-metrics.md index f1b5ad01..457ab139 100644 --- a/docs/features/prometheus-metrics.md +++ b/docs/features/prometheus-metrics.md @@ -13,8 +13,8 @@ register a `/metrics` route. Wire up `prometheus_client` exposition in your app metrics show up on your existing scrape endpoint. ```prometheus -cache_operations_total{operation="get",namespace="users",success="True",serializer="default"} 9847 -redis_cache_operations_total{operation="get",status="hit",serializer="default",namespace="users"} 9847 +cache_operations_total{operation="get",namespace="users",success="True",serializer="l1_memory"} 31022 +cache_operations_total{operation="get",namespace="users",success="True",serializer="rust"} 9847 ``` --- @@ -70,16 +70,22 @@ scrape_configs: cachekit emits the following metrics on the default `prometheus_client` registry. The names below are the **actual** series names — none carry a `cachekit_` prefix. +> The `serializer` label is the tier that served the record, not the `@cache(serializer=...)` +> preset: `rust` = L2 backend path, `l1_memory` = L1 in-memory hit; `unknown` marks a record +> emitted without the label (an instrumentation gap, not a tier). + ### Counters (always increasing) ```prometheus # Cache operations from the async/sync metrics path. # Labels: operation, namespace, success, serializer -cache_operations_total{operation="get",namespace="users",success="True",serializer="default"} +cache_operations_total{operation="get",namespace="users",success="True",serializer="rust"} +cache_operations_total{operation="get",namespace="users",success="True",serializer="l1_memory"} -# Cache operations from the backpressure/load-control path. +# Backpressure rejections from the load-control path. Emitted only when the request +# queue is full; serializer and namespace are always "" on this series. # Labels: operation, status, serializer, namespace -redis_cache_operations_total{operation="get",status="hit",serializer="default",namespace="users"} +redis_cache_operations_total{operation="backpressure",status="rejected",serializer="",namespace=""} # Decrypt/integrity failures on the read path, split by failure class. # Labels: reason — "auth_tamper" (AES-GCM auth failure, tenant mismatch, or @@ -97,20 +103,23 @@ cachekit_decrypt_failures_total{reason="auth_tamper",tier="l2"} cachekit_config_drift_reads_total{reason="encryption_disabled"} ``` -> Hits and misses are not separate series. Compute them from labels — the `success` label -> on `cache_operations_total` and the `status` label on `redis_cache_operations_total` -> distinguish hits from misses. +> Hit/miss is not exposed as a series or a label. `cache_operations_total` carries +> `operation`, `namespace`, `success`, and `serializer` — `success` is operation success +> across reads, writes, and other operations, **not** a cache hit, and there is no `hit` +> label. `redis_cache_operations_total` carries no hit/miss status either — it counts +> backpressure rejections only. For a miss signal, use `operation="set"` as a cache-write +> proxy (see [Query Examples](#query-examples)), not a hit/miss series. ### Histograms (latency and size) ```prometheus # Cache operation duration in milliseconds. # Labels: operation, namespace, serializer -cache_operation_duration_ms{operation="get",namespace="users",serializer="default"} +cache_operation_duration_ms{operation="get",namespace="users",serializer="rust"} # Cache operation payload size in bytes. # Labels: operation, namespace, serializer -cache_operation_size_bytes{operation="get",namespace="users",serializer="default"} +cache_operation_size_bytes{operation="get",namespace="users",serializer="l1_memory"} ``` ### Gauges (current state) @@ -125,18 +134,24 @@ circuit_breaker_state{namespace="users",state="open"} ## Query Examples -### Cache Hit Rate +### Operation Success Rate + +`cache_operations_total` has no hit/miss label — `success` covers reads, writes, and +other operations, so this is an operation-success rate, not a hit rate. ```promql -# Hit rate (percentage) using the success label on cache_operations_total +# Operation success rate (percentage) using the success label on cache_operations_total 100 * sum(rate(cache_operations_total{success="True"}[5m])) / sum(rate(cache_operations_total[5m])) ``` +For a miss-rate proxy, watch cache writes: `operation="set"` is recorded when a miss +writes back, but it is a write proxy — it can double-count when stats collection is on +and records nothing for a failed write — so treat it as a proxy, not an exact miss count. + ```promql -# Hit rate from the load-control path using the status label -100 * sum(rate(redis_cache_operations_total{status="hit"}[5m])) - / sum(rate(redis_cache_operations_total[5m])) +# Cache-write rate as a miss-rate proxy (see caveats above) +sum(rate(cache_operations_total{operation="set"}[5m])) ``` ### Cache Latency (P99) @@ -168,16 +183,16 @@ circuit_breaker_state ## Alerting Examples -### Alert: Low Cache Hit Rate +### Alert: Low Operation Success Rate ```yaml -- alert: LowCacheHitRate +- alert: LowCacheOperationSuccessRate expr: | 100 * sum(rate(cache_operations_total{success="True"}[5m])) / sum(rate(cache_operations_total[5m])) - < 50 # Hit rate below 50% + < 50 # Operation success below 50% (this is not a hit rate) annotations: - summary: "Cache hit rate is low (< 50%)" + summary: "Cache operation success rate is low (< 50%)" ``` ### Alert: Circuit Breaker Open @@ -274,9 +289,10 @@ A: cachekit does not expose metrics for you. Confirm your app starts the **default** registry, and that at least one decorated function has run — series are created lazily on first use. -**Q: Hit rate always 0** -A: Check that the function is actually being called and that L1/L2 caching is working. -Remember hit/miss is derived from labels (`success` / `status`), not from separate series. +**Q: Where is the hit rate?** +A: There is no hit/miss series or label. `cache_operations_total` exposes operation +success (`success`), not hits. Use `operation="set"` as a cache-write proxy for misses +(see [Query Examples](#query-examples)) — it is a proxy, not an exact count. **Q: Metrics growing unbounded** A: Prometheus retention is configurable (default 15 days). Keep label cardinality bounded —