Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -838,11 +838,16 @@ HTTP server or register a `/metrics` route; wire up `prometheus_client` expositi
names carry no `cachekit_` prefix:

- `cache_operations_total` - Operation counter. Labels: `operation`, `namespace`, `success`, `serializer`
- `redis_cache_operations_total` - Load-control operation counter. Labels: `operation`, `status`, `serializer`, `namespace`
- `redis_cache_operations_total` - Load-control rejection counter. Labels: `operation`, `status`, `serializer`, `namespace`
- `cache_operation_duration_ms` - Operation latency histogram (milliseconds). Labels: `operation`, `namespace`, `serializer`
- `cache_operation_size_bytes` - Operation payload size histogram (bytes). Labels: `operation`, `namespace`, `serializer`
- `circuit_breaker_state` - Circuit breaker state gauge (0=CLOSED, 1=OPEN, 2=HALF_OPEN). Labels: `namespace`, `state`

The `serializer` label is the tier that served the record, not the `@cache(serializer=...)`
preset: `rust` = L2 backend path, `l1_memory` = L1 in-memory hit; `unknown` marks a record
emitted without the label. `redis_cache_operations_total` is emitted only on backpressure
rejection (`operation="backpressure"`, `status="rejected"`, empty `serializer` and `namespace`).

See the [Prometheus Metrics guide](features/prometheus-metrics.md) for exposition setup,
query examples, and alerting rules.

Expand Down
9 changes: 5 additions & 4 deletions docs/features/distributed-locking.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,9 +318,10 @@ async def fetch_sensitive(x):

Lock waiters that time out log a `Failed to acquire lock for {key} after 5.0s`
warning; lock backend errors log a `Lock operation failed … executing without
lock` warning. For miss-rate monitoring (stampede detection), use the `status`
label on `redis_cache_operations_total` — see
[Prometheus Metrics](prometheus-metrics.md).
lock` warning. For miss-rate monitoring (stampede detection), watch `operation="set"` on
`cache_operations_total` as a cache-write proxy — misses write back, but it can
double-count (when stats collection is on) or miss failed writes, so treat it as a proxy,
not an exact miss count — see [Prometheus Metrics](prometheus-metrics.md).

---

Expand All @@ -335,7 +336,7 @@ A: Two things to check:
2. The backend must implement `LockableBackend` (`RedisBackend`, `CachekitIOBackend`). Check with `from cachekit.backends.base import LockableBackend; isinstance(backend, LockableBackend)`.

**Q: How do I know if stampedes are happening?**
A: Check Prometheus: a spike in `rate(redis_cache_operations_total{status="miss"}[1m])` = stampede risk. See [Prometheus Metrics](prometheus-metrics.md).
A: Check Prometheus: a spike in `rate(cache_operations_total{operation="set"}[1m])` (a cache-write proxy — misses write back) suggests stampede risk. See [Prometheus Metrics](prometheus-metrics.md).

---

Expand Down
60 changes: 38 additions & 22 deletions docs/features/prometheus-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ register a `/metrics` route. Wire up `prometheus_client` exposition in your app
metrics show up on your existing scrape endpoint.

```prometheus
cache_operations_total{operation="get",namespace="users",success="True",serializer="default"} 9847
redis_cache_operations_total{operation="get",status="hit",serializer="default",namespace="users"} 9847
cache_operations_total{operation="get",namespace="users",success="True",serializer="l1_memory"} 31022
cache_operations_total{operation="get",namespace="users",success="True",serializer="rust"} 9847
```

---
Expand Down Expand Up @@ -70,16 +70,22 @@ scrape_configs:
cachekit emits the following metrics on the default `prometheus_client` registry. The names
below are the **actual** series names — none carry a `cachekit_` prefix.

> The `serializer` label is the tier that served the record, not the `@cache(serializer=...)`
> preset: `rust` = L2 backend path, `l1_memory` = L1 in-memory hit; `unknown` marks a record
> emitted without the label (an instrumentation gap, not a tier).

### Counters (always increasing)

```prometheus
# Cache operations from the async/sync metrics path.
# Labels: operation, namespace, success, serializer
cache_operations_total{operation="get",namespace="users",success="True",serializer="default"}
cache_operations_total{operation="get",namespace="users",success="True",serializer="rust"}
cache_operations_total{operation="get",namespace="users",success="True",serializer="l1_memory"}

# Cache operations from the backpressure/load-control path.
# Backpressure rejections from the load-control path. Emitted only when the request
# queue is full; serializer and namespace are always "" on this series.
# Labels: operation, status, serializer, namespace
redis_cache_operations_total{operation="get",status="hit",serializer="default",namespace="users"}
redis_cache_operations_total{operation="backpressure",status="rejected",serializer="",namespace=""}

# Decrypt/integrity failures on the read path, split by failure class.
# Labels: reason — "auth_tamper" (AES-GCM auth failure, tenant mismatch, or
Expand All @@ -97,20 +103,23 @@ cachekit_decrypt_failures_total{reason="auth_tamper",tier="l2"}
cachekit_config_drift_reads_total{reason="encryption_disabled"}
```

> Hits and misses are not separate series. Compute them from labels — the `success` label
> on `cache_operations_total` and the `status` label on `redis_cache_operations_total`
> distinguish hits from misses.
> Hit/miss is not exposed as a series or a label. `cache_operations_total` carries
> `operation`, `namespace`, `success`, and `serializer` — `success` is operation success
> across reads, writes, and other operations, **not** a cache hit, and there is no `hit`
> label. `redis_cache_operations_total` carries no hit/miss status either — it counts
> backpressure rejections only. For a miss signal, use `operation="set"` as a cache-write
> proxy (see [Query Examples](#query-examples)), not a hit/miss series.

### Histograms (latency and size)

```prometheus
# Cache operation duration in milliseconds.
# Labels: operation, namespace, serializer
cache_operation_duration_ms{operation="get",namespace="users",serializer="default"}
cache_operation_duration_ms{operation="get",namespace="users",serializer="rust"}

# Cache operation payload size in bytes.
# Labels: operation, namespace, serializer
cache_operation_size_bytes{operation="get",namespace="users",serializer="default"}
cache_operation_size_bytes{operation="get",namespace="users",serializer="l1_memory"}
```

### Gauges (current state)
Expand All @@ -125,18 +134,24 @@ circuit_breaker_state{namespace="users",state="open"}

## Query Examples

### Cache Hit Rate
### Operation Success Rate

`cache_operations_total` has no hit/miss label — `success` covers reads, writes, and
other operations, so this is an operation-success rate, not a hit rate.

```promql
# Hit rate (percentage) using the success label on cache_operations_total
# Operation success rate (percentage) using the success label on cache_operations_total
100 * sum(rate(cache_operations_total{success="True"}[5m]))
/ sum(rate(cache_operations_total[5m]))
```

For a miss-rate proxy, watch cache writes: `operation="set"` is recorded when a miss
writes back, but it is a write proxy — it can double-count when stats collection is on
and records nothing for a failed write — so treat it as a proxy, not an exact miss count.

```promql
# Hit rate from the load-control path using the status label
100 * sum(rate(redis_cache_operations_total{status="hit"}[5m]))
/ sum(rate(redis_cache_operations_total[5m]))
# Cache-write rate as a miss-rate proxy (see caveats above)
sum(rate(cache_operations_total{operation="set"}[5m]))
```

### Cache Latency (P99)
Expand Down Expand Up @@ -168,16 +183,16 @@ circuit_breaker_state

## Alerting Examples

### Alert: Low Cache Hit Rate
### Alert: Low Operation Success Rate

```yaml
- alert: LowCacheHitRate
- alert: LowCacheOperationSuccessRate
expr: |
100 * sum(rate(cache_operations_total{success="True"}[5m]))
/ sum(rate(cache_operations_total[5m]))
< 50 # Hit rate below 50%
< 50 # Operation success below 50% (this is not a hit rate)
annotations:
summary: "Cache hit rate is low (< 50%)"
summary: "Cache operation success rate is low (< 50%)"
```

### Alert: Circuit Breaker Open
Expand Down Expand Up @@ -274,9 +289,10 @@ A: cachekit does not expose metrics for you. Confirm your app starts
the **default** registry, and that at least one decorated function has run — series are
created lazily on first use.

**Q: Hit rate always 0**
A: Check that the function is actually being called and that L1/L2 caching is working.
Remember hit/miss is derived from labels (`success` / `status`), not from separate series.
**Q: Where is the hit rate?**
A: There is no hit/miss series or label. `cache_operations_total` exposes operation
success (`success`), not hits. Use `operation="set"` as a cache-write proxy for misses
(see [Query Examples](#query-examples)) — it is a proxy, not an exact count.

**Q: Metrics growing unbounded**
A: Prometheus retention is configurable (default 15 days). Keep label cardinality bounded —
Expand Down
Loading