fix(prism): WTA emission, pre-pod screens, 8 concurrent evals - #85
Conversation
Collapse competition credits to a single Score leaf so Prism's 50% share goes to one winner; fail static/similarity cheats before Lium rent; scale orchestrator workers to 8 on prod.
|
Warning Review limit reached
Next review available in: 43 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (14)
📝 WalkthroughWalkthroughChangesThe PR moves Prism anti-cheat checks before pod provisioning, adds shared static source checks, applies winner-take-all scoring during epoch emission, and raises default concurrent evaluations from 1 to 8. Pre-pod anti-cheat screening
Winner-take-all Prism emission
Evaluation concurrency defaults
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Submission
participant Orchestrator
participant StaticChecks
participant SimilarityCheck
participant PodRental
Submission->>Orchestrator: submit architecture and training sources
Orchestrator->>StaticChecks: check source cheats and telemetry hooks
Orchestrator->>SimilarityCheck: compare architecture sources
SimilarityCheck-->>Orchestrator: pass or copied verdict
Orchestrator->>PodRental: provision pod after screening passes
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (1)
docs/runbooks/prism-enable-lium-and-emission.md (1)
9-10: 🚀 Performance & Scalability | 🔵 TrivialAdd a capacity preflight before enabling eight workers.
Line [9] raises a billable production default. Add a preflight that records Lium lease-pool capacity and verifies that it can support
PRISM_MAX_CONCURRENT_EVALS=8before rollout.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/runbooks/prism-enable-lium-and-emission.md` around lines 9 - 10, Add a preflight step before enabling the documented production default of PRISM_MAX_CONCURRENT_EVALS=8 that records the available Lium lease-pool capacity and verifies it is at least eight concurrent evaluations; require this check to pass before rollout and retain the existing guidance to reduce the worker count when capacity is insufficient.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/challenge-agentic/src/static_checks.rs`:
- Around line 23-26: Format the static_source_cheat function signature in
crates/challenge-agentic/src/static_checks.rs using cargo fmt --all, then verify
the CI-equivalent formatting gate passes.
- Around line 50-56: Update training_has_telemetry_hooks to parse training_py
with the existing AST support and require executable report and
finish_evaluation call nodes on an allowed telemetry object, rather than
matching raw substrings that comments can spoof. Preserve the accepted telemetry
access forms, and add a regression test demonstrating that comment-only or
non-executable text does not satisfy the check.
In `@crates/prism-challenge/src/orchestrator.rs`:
- Around line 545-558: Handle the result of store.apply in the rejection path
instead of discarding it. In the surrounding method, propagate or retry any
state-write error and only update the gating record to Rejected after the
rejection row has persisted successfully; preserve the existing rejection patch
and StageEvent behavior.
In `@crates/prism-emit/src/lib.rs`:
- Around line 211-212: Filter the map returned by
prism_registry::competition_scores to hotkeys in expected.participants before
passing it to prism_registry::apply_wta, while preserving the existing WTA and
signed-set flow. Add a test covering an expected lower-scoring hotkey alongside
a higher-scoring outside hotkey, verifying the expected participant remains
eligible and receives the correct result.
In `@crates/prism-pipeline/src/config.rs`:
- Around line 12-13: Connect PrismConfig’s max_concurrent_evals value from
PrismConfig::production()/live_smoke() through the CLI/default path used by
spawn_orchestrator, replacing the unrelated Cli::max_concurrent_evals source.
Preserve explicit configuration overrides while ensuring the documented profile
default of 8 is applied when no override is provided.
In `@docker-compose.yml`:
- Line 208: Remove the duplicate PRISM_MAX_CONCURRENT_EVALS service-level
entries from docker-compose.yml (208-208) and deploy/compose/env-prod.yml
(52-53). Preserve the CLI default and deploy/env/prism-challenge.env as the
supported configuration sources so operator overrides are not overwritten.
In `@docs/PRISM.md`:
- Around line 44-48: Update the opening workflow overview in PRISM.md to match
the state machine: run the copy gate, static telemetry/cheat screen, and
similarity screen before pod provisioning, then perform the LLM/agentic review
after post-pod metrics and quality checks. Remove or revise any wording that
places LLM review before similarity or requires telemetry hooks at that earlier
stage.
---
Nitpick comments:
In `@docs/runbooks/prism-enable-lium-and-emission.md`:
- Around line 9-10: Add a preflight step before enabling the documented
production default of PRISM_MAX_CONCURRENT_EVALS=8 that records the available
Lium lease-pool capacity and verifies it is at least eight concurrent
evaluations; require this check to pass before rollout and retain the existing
guidance to reduce the worker count when capacity is insufficient.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 4fc66d32-37dc-4a61-9f2b-c880df8293e6
📒 Files selected for processing (21)
bins/prism-challenge/src/main.rscrates/challenge-agentic/src/lib.rscrates/challenge-agentic/src/sim.rscrates/challenge-agentic/src/static_checks.rscrates/prism-challenge/src/orchestrator.rscrates/prism-challenge/tests/cheat_arch_copy.rscrates/prism-challenge/tests/cheat_metrics.rscrates/prism-challenge/tests/copy_gate.rscrates/prism-emit/src/lib.rscrates/prism-emit/tests/epoch_semantics.rscrates/prism-pipeline/src/config.rscrates/prism-registry/src/competition.rscrates/prism-registry/src/lib.rsdeploy/compose/env-prod.ymldeploy/env/prism-challenge.env.exampledocker-compose.ymldocs/COMPLETENESS.mddocs/PRISM.mddocs/evidence/prism-wta-2026-08-08.mddocs/external-miner/prism.mddocs/runbooks/prism-enable-lium-and-emission.md
| let _ = self | ||
| .store | ||
| .apply( | ||
| &row.id, | ||
| &StatePatch { | ||
| status: Some(Stage::Rejected), | ||
| final_score: Some(FinalScore::Score(0)), | ||
| similarity: Some(similarity), | ||
| error_detail: Some(format!( | ||
| "copy gate: architecture clones {} (bps={})", | ||
| hit.nearest_id, hit.similarity_bps | ||
| )), | ||
| similarity, | ||
| error_detail: Some(error_detail), | ||
| ..StatePatch::default() | ||
| }, | ||
| Some(&StageEvent { | ||
| stage: Stage::Rejected, | ||
| detail: Some(serde_json::json!({ | ||
| "gate": "copy_created_at", | ||
| "nearest_id": hit.nearest_id, | ||
| "similarity_bps": hit.similarity_bps, | ||
| "byte_identical": hit.byte_identical, | ||
| })), | ||
| detail, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Do not ignore the rejection state-write failure.
If store.apply fails, this method still returns as handled and can set the gating record to Rejected. The submission row can then remain non-terminal while the miner cannot retry.
Propagate or retry the store error. Set the gating terminal state only after the rejection row persists.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/prism-challenge/src/orchestrator.rs` around lines 545 - 558, Handle
the result of store.apply in the rejection path instead of discarding it. In the
surrounding method, propagate or retry any state-write error and only update the
gating record to Rejected after the rejection row has persisted successfully;
preserve the existing rejection patch and StageEvent behavior.
| let by_miner = | ||
| prism_registry::apply_wta(prism_registry::competition_scores(batch, arch_owners)); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Filter to expected participants before WTA.
competition_scores can contain a hotkey that is absent from expected.participants. If that hotkey has the highest score, apply_wta selects it, and lines 215-226 omit it from the signed set. Every eligible lower-scoring participant then receives Score(0).
Filter the competition map to expected hotkeys before calling apply_wta. Add a test with one expected lower-scoring hotkey and one higher-scoring outside hotkey.
Proposed fix
+ let expected_hotkeys: BTreeSet<String> = expected
+ .participants
+ .iter()
+ .map(|participant| hex::encode(participant.hotkey))
+ .collect();
let by_miner =
- prism_registry::apply_wta(prism_registry::competition_scores(batch, arch_owners));
+ prism_registry::apply_wta(
+ prism_registry::competition_scores(batch, arch_owners)
+ .into_iter()
+ .filter(|(hotkey, _)| expected_hotkeys.contains(hotkey))
+ .collect(),
+ );📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| let by_miner = | |
| prism_registry::apply_wta(prism_registry::competition_scores(batch, arch_owners)); | |
| let expected_hotkeys: BTreeSet<String> = expected | |
| .participants | |
| .iter() | |
| .map(|participant| hex::encode(participant.hotkey)) | |
| .collect(); | |
| let by_miner = | |
| prism_registry::apply_wta( | |
| prism_registry::competition_scores(batch, arch_owners) | |
| .into_iter() | |
| .filter(|(hotkey, _)| expected_hotkeys.contains(hotkey)) | |
| .collect(), | |
| ); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/prism-emit/src/lib.rs` around lines 211 - 212, Filter the map returned
by prism_registry::competition_scores to hotkeys in expected.participants before
passing it to prism_registry::apply_wta, while preserving the existing WTA and
signed-set flow. Add a test covering an expected lower-scoring hotkey alongside
a higher-scoring outside hotkey, verifying the expected participant remains
eligible and receives the correct result.
| /// Global concurrent Lium evals (horizontal scale; prod default 8). | ||
| pub max_concurrent_evals: u32, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline crates/prism-pipeline/src/config.rs \
--items all --type function --match 'production|live_smoke'
rg -n -P '\bmax_concurrent_evals\b|PrismConfig::(production|live_smoke)|spawn_orchestrator' \
bins crates testsRepository: BaseIntelligence/base
Length of output: 1205
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- crates/prism-pipeline/src/config.rs ---\n'
sed -n '1,90p' crates/prism-pipeline/src/config.rs
printf '\n--- bins/prism-challenge/src/main.rs relevant sections ---\n'
sed -n '1,110p' bins/prism-challenge/src/main.rs
sed -n '430,530p' bins/prism-challenge/src/main.rs
printf '\n--- Config and PrismConfig references ---\n'
rg -n -P 'PrismConfig|config\.rs:.*PrismConfig|\.production\(|\.live_smoke\(|PrismConfig::(production|live_smoke)' bins crates --glob '!**/.git/**'Repository: BaseIntelligence/base
Length of output: 250
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- crates/prism-pipeline/src/config.rs ---'
sed -n '1,90p' crates/prism-pipeline/src/config.rs
echo
echo '--- bins/prism-challenge/src/main.rs relevant sections ---'
sed -n '1,110p' bins/prism-challenge/src/main.rs
sed -n '430,530p' bins/prism-challenge/src/main.rs
echo
echo '--- Config and PrismConfig references ---'
rg -n -P 'PrismConfig|config\.rs:.*PrismConfig|\.production\(|\.live_smoke\(|PrismConfig::(production|live_smoke)' bins crates --glob '!**/.git/**'Repository: BaseIntelligence/base
Length of output: 11179
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- prism-pipeline pipeline.rs ---'
sed -n '1,220p' crates/prism-pipeline/src/pipeline.rs
echo
echo '--- prism-challenge lib references ---'
sed -n '1,120p' crates/prism-challenge/src/lib.rs
echo
echo '--- all PrismConfig::production/livesim/sim call sites with context ---'
rg -n -C 3 'PrismConfig::(production|live_smoke|sim)\(\)' crates/prism-challenge crates/prism-pipeline --glob '!**/.git/**'Repository: BaseIntelligence/base
Length of output: 10391
Avoid diverging concurrency defaults.
PrismConfig::production()/live_smoke() set max_concurrent_evals = 8, but spawn_orchestrator reads Cli::max_concurrent_evals instead of the Prism profile. Connect the profile to the CLI/default path so the documented production default is not silently masked by an unrelated CLI/env value.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/prism-pipeline/src/config.rs` around lines 12 - 13, Connect
PrismConfig’s max_concurrent_evals value from
PrismConfig::production()/live_smoke() through the CLI/default path used by
spawn_orchestrator, replacing the unrelated Cli::max_concurrent_evals source.
Preserve explicit configuration overrides while ensuring the documented profile
default of 8 is applied when no override is provided.
| OPENROUTER_API_KEY_FILE: /run/base/openrouter/api_key | ||
| BASE_CHALLENGE_GATEWAY_ENDPOINT: ${BASE_CHALLENGE_GATEWAY_ENDPOINT:-http://gateway:8080} | ||
| PRISM_MAX_CONCURRENT_EVALS: "${PRISM_MAX_CONCURRENT_EVALS:-1}" | ||
| PRISM_MAX_CONCURRENT_EVALS: "${PRISM_MAX_CONCURRENT_EVALS:-8}" |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Files matching compose/env:\n'
git ls-files | rg '(^|/)(docker-compose\.yml|compose\.ya?ml|env-prod\.ya?ml|prism-challenge\.env|compose|compose\.)' || true
printf '\ndocker-compose.yml around relevant lines / service env declarations:\n'
if [ -f docker-compose.yml ]; then
nl -ba docker-compose.yml | sed -n '180,225p'
printf '\nOccurrences of PRISM_MAX_CONCURRENT_EVALS in docker-compose.yml:\n'
rg -n 'PRISM_MAX_CONCURRENT_EVALS|deploy/env/prism-challenge\.env|env_file' docker-compose.yml
fi
printf '\ndeploy/compose/env-prod.yml around relevant lines:\n'
if [ -f deploy/compose/env-prod.yml ]; then
nl -ba deploy/compose/env-prod.yml | sed -n '1,120p'
printf '\nOccurrences of PRISM_MAX_CONCURRENT_EVALS in deploy/compose/env-prod.yml:\n'
rg -n 'PRISM_MAX_CONCURRENT_EVALS|deploy/env/prism-challenge\.env|env_file' deploy/compose/env-prod.yml
fi
printf '\nOccurrences of deploy/env/prism-challenge.env across repo:\n'
rg -n 'deploy/env/prism-challenge\.env|prism-challenge\.env' . || trueRepository: BaseIntelligence/base
Length of output: 665
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'docker-compose.yml relevant sections:\n'
awk 'NR>=170 && NR<=230 {printf "%6d %s\n", NR, $0}' docker-compose.yml
printf '\nOccurrences in docker-compose.yml:\n'
grep -nE 'PRISM_MAX_CONCURRENT_EVALS|deploy/env/prism-challenge\.env|env_file' docker-compose.yml || true
printf '\ndeploy/compose/env-prod.yml relevant sections:\n'
awk 'NR>=40 && NR<=65 {printf "%6d %s\n", NR, $0}' deploy/compose/env-prod.yml
printf '\nOccurrences in deploy/compose/env-prod.yml:\n'
grep -nE 'PRISM_MAX_CONCURRENT_EVALS|deploy/env/prism-challenge\.env|env_file' deploy/compose/env-prod.yml || true
printf '\nprism-challenge env files:\n'
for f in deploy/env/prism-challenge.env.example deploy/env/prism-challenge.env deploy/compose/env-local.yml deploy/compose/env-staging.yml deploy/compose/role-master.yml deploy/compose/role-validator.yml; do
if [ -f "$f" ]; then
printf '\n=== %s ===\n' "$f"
grep -n 'PRISM_MAX_CONCURRENT_EVALS' "$f" || true
fi
done
printf '\nPRISM_MAX_CONCURRENT_EVALS all occurrences:\n'
grep -RIn 'PRISM_MAX_CONCURRENT_EVALS' . || trueRepository: BaseIntelligence/base
Length of output: 6519
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'docker-compose service list / includes / prism-challenge all occurrences:\n'
grep -nE '^[[:space:]]*(includes|services|prism-challenge|BASE_PRISM|max_concurrent|PRISM_MAX_CONCURRENT|PRISM_MAX)' docker-compose.yml || true
printf '\ndeploy/compose files top-level composition references:\n'
for f in deploy/compose/*.yml; do
[ -f "$f" ] || continue
printf '\n=== %s ===\n' "$f"
grep -nE '^[[:space:]]*(includes|extends|services|prism-challenge|max_concurrent|PRISM_MAX_CONCURRENT)' "$f" || true
doneRepository: BaseIntelligence/base
Length of output: 786
Remove the duplicate PRISM_MAX_CONCURRENT_EVALS entries.
docker-compose.yml and deploy/compose/env-prod.yml both define PRISM_MAX_CONCURRENT_EVALS in environment, and Compose resolves in-file environment after env_file, so these values override operator settings from deploy/env/prism-challenge.env. Keep the CLI default plus deploy/env/prism-challenge.env as the documented dial-down path, and remove the duplicate service-level entry.
📍 Affects 2 files
docker-compose.yml#L208-L208(this comment)deploy/compose/env-prod.yml#L52-L53
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docker-compose.yml` at line 208, Remove the duplicate
PRISM_MAX_CONCURRENT_EVALS service-level entries from docker-compose.yml
(208-208) and deploy/compose/env-prod.yml (52-53). Preserve the CLI default and
deploy/env/prism-challenge.env as the supported configuration sources so
operator overrides are not overwritten.
Source: Coding guidelines
| Queued --> Rejected: pre-pod screens (copy gate / static cheat / similarity) | ||
| Queued --> Provisioning: worker claims + pre-pod screens pass | ||
| Provisioning --> Running: pod SSH + harness up | ||
| Running --> Reviewing: METRICS_JSON collected | ||
| Reviewing --> AgenticReview: arch-only similarity + quality | ||
| Reviewing --> AgenticReview: quality + post-pod agentic |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Align the opening workflow description with the state machine.
The opening overview still says that LLM review occurs before similarity and enforces telemetry hooks. The changed state machine now runs static telemetry and similarity screens before pod provisioning.
Update the overview so the normative document has one screening order.
As per coding guidelines, treat normative documentation as the source of truth for contracts and operations.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/PRISM.md` around lines 44 - 48, Update the opening workflow overview in
PRISM.md to match the state machine: run the copy gate, static telemetry/cheat
screen, and similarity screen before pod provisioning, then perform the
LLM/agentic review after post-pod metrics and quality checks. Remove or revise
any wording that places LLM review before similarity or requires telemetry hooks
at that earlier stage.
Source: Coding guidelines
Keep prism-challenge and challenge-agentic under the 1500 LOC gate while preserving pre-pod static screens and GatewayClient via challenge-common.
1 participant
Summary
apply_wtaafter competition credits so only one hotkey gets Prism's 50% share (prod had 3 positive Score leaves → soft allocation).METRICS_JSON/telemetry hooks → AST similarity.PRISM_MAX_CONCURRENT_EVALSto 8 (CLI/compose defaults +env-prod.yml); orchestrator spawns N semaphore-gated workers.Evidence (before)
Live
epoch 24369prism leaves: champion5HK8…177155, challenger5CY9…111595, arch-owner5HHL…111595 → proportional weights 0.221/0.139/0.139 of subnet (plus design). Seedocs/evidence/prism-wta-2026-08-08.md.Test plan
cargo test -p prism-registry -p prism-emit -p challenge-agentic -p prism-challengePRISM_MAX_CONCURRENT_EVALS=8/v1/weights/latestSummary by CodeRabbit
New Features
Scoring Improvements
Bug Fixes