Lazyfying - #239
Open
nikbpetrov wants to merge 2 commits into
Open
Lazyfying#239nikbpetrov wants to merge 2 commits into
nikbpetrov wants to merge 2 commits into
Conversation
nikbpetrov
force-pushed
the
refactor/lazy-imports
branch
from
June 30, 2026 17:41
0a2dc33 to
d0914f1
Compare
nikbpetrov
force-pushed
the
refactor/lazy-imports
branch
from
September 10, 2026 02:04
7de71bd to
ee90d58
Compare
houtanb
reviewed
Sep 10, 2026
| # Optional reproducibility: when QUESTION_SET_SEED is set, thread a single RandomState through | ||
| # both sampling passes so the published set is deterministic. Unset (the default) preserves the | ||
| # historical unseeded behaviour. | ||
| seed_env = os.getenv("QUESTION_SET_SEED") |
Member
There was a problem hiding this comment.
should we also check that seed_env is None when we're in prod to ensure it isn't accidentally ever set in deployment?
nikbpetrov
force-pushed
the
refactor/lazy-imports
branch
from
September 22, 2026 10:45
ee90d58 to
5c4c4ec
Compare
houtanb
force-pushed
the
refactor/lazy-imports
branch
2 times, most recently
from
September 24, 2026 12:14
7ac2754 to
69da6cf
Compare
…aderboard bootstrapping Add an optional `RANDOM_SEED` environment variable, exposed as `env.RANDOM_SEED` (`int`, `None` when unset). When set, it seeds the question sampling in `create_question_set` (the human and LLM samplers, threading a single `np.random.RandomState` so successive draws decorrelate) and the question-level bootstrap behind the simulated leaderboards, making both testable and reproducible. It is for testing only: deployments leave it unset, so production behaviour stays unseeded, and a test ensures the root `Makefile` never ships it to Cloud Run.
Importing `helpers.keys` used to fetch every secret from Secret Manager at module load, so any module that imported it, even indirectly, needed GCP credentials and paid for network calls it might never use. Secrets are now listed in the `_SECRET_NAMES` set and resolved through a module `__getattr__` (PEP 562) on first attribute access, then memoized; the attribute name is the secret name. Callers keep using `keys.API_*` unchanged.
nikbpetrov
force-pushed
the
refactor/lazy-imports
branch
from
September 25, 2026 09:34
69da6cf to
803d0a4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
meant to be merged as separate commits, just grouped here in the same PR
7de71bd is not lazy-related, sorry for sneaking it in, but it's a 1-line print -> logger.debug, change (preps the testing framework work)