home-mixer: lower post-candidate cache compression from zstd 6 to zstd 1 (5.9x less CPU, same size)

[gauss2302]

Lowers ZSTD_COMPRESSION_LEVEL in RedisPostCandidateCacheSideEffect from 6 to 1, and adds a regression test pinning that the cache stays readable across a level change.

Net effect: 70.8% less CPU in this side effect, and 4.4% fewer bytes on the wire.

Why

RedisPostCandidateCacheSideEffect runs on every For You request not already served from cache (enable = is_prod() && !query.has_cached_posts). It serializes up to MaxPostsToCache = 750 PostCandidates to JSON and zstd-compresses them inline on the request path.

That payload is ~2 MB and unusually repetitive: 750 records sharing one 54-field schema, 45 of those Option, so field-name scaffolding repeats every ~2.7 KB. zstd's fast strategy already captures nearly all of it.

Compression was 84% of this side effect's CPU. Serialize-and-compress, per request:

	time	on the wire
serde_json alone	1.86 ms	2,005,304 B
+ zstd 6 (current)	11.43 ms	373,127 B
+ zstd 1 (this change)	3.33 ms	356,580 B

Level sweep on the same payload:

level	bytes	ms	vs level 6
1	356,580	1.83	−4.4% size, 5.2× faster
3	405,679	3.55	+8.7% size, 2.7× faster
6	373,127	9.56	(current)
9	356,381	13.98	−4.5% size, 0.7×

A full 1–9 sweep straight through libzstd (same payload, outside the Rust harness) shows the same shape: levels 2, 4 and 5 all land between these, and 2–4 are each both slower and larger than level 1.

Since the entry only lives for REDIS_TTL_SECONDS = 180, request-path CPU dominates the value of a marginally smaller blob — but here level 1 is smaller as well as faster, so there is no tradeoff to make.

Why level 1 and not level 3

Compressed size is not monotone in level on this payload. Levels 2–4 are both slower and larger than level 1; level 3 is in fact the worst ratio on the whole scale, 8.7% larger than level 6 while still costing ~2× level 1's CPU.

Because that is surprising, I checked it is not an artifact of the reconstructed text distribution. Varying only how tweet_text is generated:

tweet_text model	level 1 size vs level 6	level 1 CPU vs level 6
24-word vocabulary	−6.0%	5.7× faster
2000-word vocabulary (natural language)	+2.0%	7.7× faster
random base62 (worst case)	−2.2%	5.7× faster

Reproduced on two independent seeds. Text entropy barely moves the result because post text is only ~15% of the payload; the schema scaffolding dominates.

Compatibility

No cache key version bump and no coordinated rollout:

• zstd frames are self-describing, so zstd::decode_all reads a level-6 frame and a level-1 frame with the same call.
• redis_client::cached_posts_key does not encode the compression level.
• Entries in flight during a rolling deploy decode on hosts running either version.
• Decompression is level-agnostic (0.71 ms for a level-6 frame vs 0.75 ms for a level-3 frame — within noise), so CachedPostsQueryHydrator is unaffected.

The new test payload_written_at_any_zstd_level_round_trips pins this across levels 1/3/6/9 so the constant stays freely retunable.

How this was measured

home-mixer/ ships no Cargo.toml and depends on unpublished xai_* crates, so it cannot be built from this repository and I could not benchmark the side effect in place. Instead the payload was reconstructed field-for-field from home-mixer/models/candidate.rs — all 54 fields in declaration order, 45 Option, the nested 26-head PhoenixScores — and driven through the real serde_json and zstd crates. The population is shaped like what get_candidates_to_cache actually caches: only the first TOP_K_CANDIDATES_TO_SELECT = 50 candidates carry post-selection-hydrated fields such as brand_safety_verdict and tweet_type_metrics; the other 700 carry None there.

Median of 25 runs after 5 warmup runs, release build, Apple M-series. Happy to re-run in-tree if you have a harness that can link home-mixer; the measurement code is available on request.

Considered and rejected: skip_serializing_if on PostCandidate

I also prototyped #[serde(skip_serializing_if = "Option::is_none")] across the 45 Option fields. Two findings argue against it:

It buys almost nothing on top of this change. It shrinks the JSON 22% (2.00 MB → 1.56 MB), but on the hot path that is 3.33 ms → 3.03 ms — 0.30 ms, or 2.6% of the original 11.43 ms. Nearly all of the win in this side effect was the compression level, not the payload size.

The naive version silently breaks the read path. serde substitutes None for a missing Option, but it does not substitute an empty Vec or String. tweet_text, ancestors, tombstone_ancestor_ids and ancestor_users have no #[serde(default)] in candidate.rs, so omitting them when empty makes an old reader fail outright with missing field 'ancestors'. Landing this safely would need a two-phase deploy: first a release that adds #[serde(default)] to those fields, then a later one that starts omitting them. That is a lot of ceremony for 0.30 ms.

Raising it in case the tradeoff looks different with production data.

Aside, unrelated to this change

While building the compatibility test I found that serde_json's default float parser is not bit-exact: it writes the correct shortest representation but reads back a value 1 ULP away for 13% of random f64s (std's str::parse::<f64> is exact; serde_json's float_roundtrip feature fixes it). So Phoenix scores that go through this Redis cache come back marginally different from what was computed on a cached-posts request. The relative error is ~1e-16 and I would not change anything for it — noting it only because it is a real property of the current cache path and it is easy to trip over when writing an equality test against it.

Risk

Low. One constant, no format change, no API change, no behaviour change on the read path. Worst case is more Redis memory per slate if production data compresses less like the reconstruction than expected — though the measurement says level 1 is smaller, not larger. Either way it is visible immediately in the existing compressed_size field of the tracing::debug! at the end of side_effect.

---

🤖 Generated with Claude Code