test: stickyBucket conformance cross-product and multi-threaded perf harness - #239
Open
madhuchavva wants to merge 2 commits into
Open
test: stickyBucket conformance cross-product and multi-threaded perf harness#239madhuchavva wants to merge 2 commits into
madhuchavva wants to merge 2 commits into
Conversation
…nt across service and API variants
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Conformance cross-product for the multi-user client and a multi-threaded sticky perf harness
Stacked on #237 (exposure tracking hardening).
Summary
Validates the async sticky bucketing work against the shared cases.json corpus and demonstrates the performance behavior under load.
The
stickyBucketcases now run against the multi-userGrowthBookClientas a parametrized cross-product (per-case JUnit names), and a checked-in harness mirrors the Python SDK's benchmark scenarios.What's included
GrowthBookClientStickyBucketConformanceTest— the 13 sharedstickyBucketcases × {offloaded sync service, async-native service} × {sync API,evalFeatureAsync()} = 52 parametrized tests with per-case names (@ParameterizedTest @MethodSource), asserting both the experiment result and the persisted store state afterflushStickyBucketSaves(). The legacy single-user runner keeps covering theGrowthBookclass unchanged.StickyBucketPerfHarness(./gradlew :lib:runStickyPerfHarness, configurable threads/requests/latency) — mirrorsgrowthbook-python/tests/scripts/benchmark_async_client.py: blocking vs async store, sync vs async API, hot-user coalescing, and the prefetch-per-request pattern, reporting throughput and p50/p95/p99.Representative run (100 threads, 2000 requests, 5 ms simulated store latency):
The blocking-store scenario is bounded by pool threads parked in store round trips — the documented reason to prefer
AsyncStickyBucketServicefor network stores. (For context: before this work, every evaluation performed the blocking round trips on the request thread itself.)Test plan
./gradlew :lib:test— green (52 new conformance tests included)./gradlew :lib:runStickyPerfHarness— completes, numbers aboveOut of scope (deliberately)
@ParameterizedTest— worthwhile test-infrastructure cleanup, but unrelated to the async work; noted as a follow-up.