Skip to content

test: stickyBucket conformance cross-product and multi-threaded perf harness - #239

Open
madhuchavva wants to merge 2 commits into
mc/callback-dedup-hardeningfrom
mc/conformance-and-perf
Open

test: stickyBucket conformance cross-product and multi-threaded perf harness#239
madhuchavva wants to merge 2 commits into
mc/callback-dedup-hardeningfrom
mc/conformance-and-perf

Conversation

@madhuchavva

@madhuchavva madhuchavva commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Conformance cross-product for the multi-user client and a multi-threaded sticky perf harness

Stacked on #237 (exposure tracking hardening).

Summary

Validates the async sticky bucketing work against the shared cases.json corpus and demonstrates the performance behavior under load.

The stickyBucket cases now run against the multi-user GrowthBookClient as a parametrized cross-product (per-case JUnit names), and a checked-in harness mirrors the Python SDK's benchmark scenarios.

What's included

  • GrowthBookClientStickyBucketConformanceTest — the 13 shared stickyBucket cases × {offloaded sync service, async-native service} × {sync API, evalFeatureAsync()} = 52 parametrized tests with per-case names (@ParameterizedTest @MethodSource), asserting both the experiment result and the persisted store state after flushStickyBucketSaves(). The legacy single-user runner keeps covering the GrowthBook class unchanged.
  • StickyBucketPerfHarness (./gradlew :lib:runStickyPerfHarness, configurable threads/requests/latency) — mirrors growthbook-python/tests/scripts/benchmark_async_client.py: blocking vs async store, sync vs async API, hot-user coalescing, and the prefetch-per-request pattern, reporting throughput and p50/p95/p99.

Representative run (100 threads, 2000 requests, 5 ms simulated store latency):

Scenario Throughput p50
blocking store / sync API / distinct users ~1,430 req/s 70 ms
async store / sync API / distinct users ~14,550 req/s 6.7 ms
async store / sync API / hot user (coalesced) ~13,290 req/s 7.5 ms
async store / async API / distinct users ~14,190 req/s 7.0 ms
async store / prefetch + 10 evals per request ~15,680 req/s (~157k evals/s) 5.4 ms

The blocking-store scenario is bounded by pool threads parked in store round trips — the documented reason to prefer AsyncStickyBucketService for network stores. (For context: before this work, every evaluation performed the blocking round trips on the request thread itself.)

Test plan

  • Full suite ./gradlew :lib:test — green (52 new conformance tests included)
  • ./gradlew :lib:runStickyPerfHarness — completes, numbers above

Out of scope (deliberately)

  • Converting the remaining hand-rolled for-loop conformance suites (evalCondition, run, feature, ...) to @ParameterizedTest — worthwhile test-infrastructure cleanup, but unrelated to the async work; noted as a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant