Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions crates/challenge-agentic/src/prompts.rs
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
//! System prompt for the agentic anti-cheat loop.

/// Prompt version stamp (audit).
pub const AGENTIC_PROMPT_VERSION: &str = "agentic_v1";
pub const AGENTIC_PROMPT_VERSION: &str = "agentic_v2";

/// Prism-specific anti-cheat rules (metrics / eval short-circuit / AST copy /
/// telemetry-hook contract).
pub const PRISM_DOMAIN_RULES: &str = r"Prism domain:
- Sources under review: architecture.py + training.py (miner code only).
- Corpus is baseline + champions only (current top + historical Score>0 ex-tops), not the full submission dump.
- Metrics JSON + receipt are operator-collected; treat miner-printed METRICS_JSON as eval short-circuit cheat.
- Telemetry contract (recipe >= 1.1.0): training.py MUST import prism_telemetry and call prism_telemetry.report(loss=..., step=..., ...) during training plus prism_telemetry.finish_evaluation() to end the eval. Missing hooks = contract violation -> cheat (missing_telemetry_hooks).
- Cheat if: near-identical AST/byte copy of a corpus architecture.py (training.py is exempt from similarity — the same training script on a different architecture is fine); hardcoded METRICS_JSON in miner sources; bpb impossible vs tokens_seen/wall_clock (e.g. bpb<<1 with ~0 tokens, or tokens_seen=0 with a finite bpb).
- suspicious: strong structural overlap or inconsistent metrics without a slam-dunk forge.
- Cheat if: near-identical AST/byte copy of a corpus architecture.py (AST similarity_bps >= 9500; training.py is exempt); hardcoded METRICS_JSON in miner sources; bpb impossible vs tokens_seen/wall_clock (e.g. bpb<<1 with ~0 tokens, or tokens_seen=0 with a finite bpb); eval peeking / telemetry bypass.
- NEVER treat standard modern LM components as plagiarism: RMSNorm, LayerNorm, RoPE/ALiBi, SwiGLU/GeGLU, gated residual, parallel residual, GQA/MQA, Pre-Norm/Post-Norm. Those are public recipe ingredients, not cheats.
- suspicious: only for strong unique structural overlap with a champion (AST >= 8500) or inconsistent metrics without a slam-dunk forge. Below AST 8500 with no other cheat signal → clean.
- Quality/coherence of the model is NOT your job — only anti-cheat.";

/// Design-specific anti-cheat rules (harness / pages / sanitize).
Expand Down Expand Up @@ -46,8 +48,10 @@ Workflow:

Verdict policy:
- clean: original enough; empty cheat_codes
- suspicious: strong overlap / odd metrics — still Score(0)
- cheat: clear copy / forgery — Score(0); set cheat_codes + nearest_id + similarity_bps
- suspicious: unique structural overlap with a corpus entry (AST band) or odd metrics — still Score(0)
- cheat: clear copy / forgery / eval short-circuit — Score(0); set cheat_codes + nearest_id + similarity_bps

Do NOT put RMSNorm / RoPE / SwiGLU / LayerNorm / gated or parallel residual in rationale as copy evidence.

AST similarity_bps thresholds (must match SimAgent; use ast_diff_nearest):
- >= 9500 → cheat (ast_architecture_copy or near_identical_harness_copy)
Expand Down
20 changes: 20 additions & 0 deletions crates/db/src/prism_store.rs
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,26 @@ pub async fn list_prism_submissions(
Ok(rows)
}

/// Champion corpus: historical Score>0 WTA/leaf winners (current top + ex-tops).
///
/// # Errors
/// SQL error.
pub async fn list_prism_champions(
pool: &PgPool,
limit: i64,
) -> Result<Vec<PrismSubmissionRow>, DbError> {
let q = format!(
"SELECT {COLS} FROM prism_submission \
WHERE kind = 'score' AND score > 0 \
ORDER BY created_at DESC LIMIT $1"
);
let rows = sqlx::query_as::<_, PrismSubmissionRow>(&q)
.bind(limit)
.fetch_all(pool)
.await?;
Ok(rows)
}

/// Ascending event journal for one row.
///
/// # Errors
Expand Down
2 changes: 1 addition & 1 deletion crates/prism-challenge/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ async fn post_precheck(
&req.architecture_py,
now_ms(),
);
let recent = st.store.list(None, None, 64).await.unwrap_or_default();
let recent = st.store.list_champions(64).await.unwrap_or_default();
let result = evaluate_copy_precheck(&candidate, &recent, quota);
Json(precheck_json(&result)).into_response()
}
Expand Down
22 changes: 13 additions & 9 deletions crates/prism-challenge/src/orchestrator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ pub struct OrchestratorConfig {
pub emit_poll: Duration,
/// Rent attempt budget before `failed`.
pub max_attempts: u32,
/// Similarity corpus size (recent submissions + baseline).
/// Similarity / agentic corpus size (champions + baseline).
pub similarity_corpus_limit: u32,
/// Stuck sweep grace (seconds). Must exceed max healthy wall-clock of a
/// live worker hold: `PRISM_SSH_RUNNING_TIMEOUT` (≤15m) + train cap (6h) +
Expand Down Expand Up @@ -457,10 +457,10 @@ impl<C: ChainClient + Send> Orchestrator<C> {
return None;
}
};
if matches!(
similarity.kind,
prism_review::SimilarityKind::Copied | prism_review::SimilarityKind::Suspicious
) {
// Only hard-reject LLM `Copied`. `Suspicious` is advisory — agentic
// (post-pod, AST-thresholded) is the primary judge. Generic LM tropes
// are coerced to Original in prism-review parsers.
if matches!(similarity.kind, prism_review::SimilarityKind::Copied) {
let detail = format!("pre-pod similarity: {:?}", similarity.kind);
self.reject_pre_pod(row, Some(similarity), None, detail)
.await;
Expand All @@ -472,7 +472,7 @@ impl<C: ChainClient + Send> Orchestrator<C> {
/// Pre-LLM copy gate on `architecture.py`. Returns `true` when the row was
/// finalized terminal `rejected` (caller must stop processing).
///
/// The corpus is recent submissions (any status — prior art is prior art)
/// The corpus is **champions** (Score>0 current top + historical ex-tops),
/// ordered by store `created_at`; the published baseline is exempt by id
/// prefix inside [`copy_gate`]. Ties / unknown timestamps fall through to
/// the LLM similarity review. Training-only rows (`arch_id` set) skip the
Expand All @@ -481,7 +481,11 @@ impl<C: ChainClient + Send> Orchestrator<C> {
if row.arch_id.is_some() {
return false;
}
let recent = self.store.list(None, None, 64).await.unwrap_or_default();
let recent = self
.store
.list_champions(self.cfg.similarity_corpus_limit.max(64))
.await
.unwrap_or_default();
let corpus = gate_corpus_from_rows(row, &recent);
let Some(hit) = copy_gate(&row.architecture_py, row.created_at_ms, &corpus) else {
return false;
Expand Down Expand Up @@ -744,7 +748,7 @@ impl<C: ChainClient + Send> Orchestrator<C> {
};
let recent = self
.store
.list(Some("terminated"), None, self.cfg.similarity_corpus_limit)
.list_champions(self.cfg.similarity_corpus_limit)
.await
.unwrap_or_default();
// Training-only rows: drop the referenced registry arch from the
Expand Down Expand Up @@ -811,7 +815,7 @@ impl<C: ChainClient + Send> Orchestrator<C> {
async fn similarity_corpus(&self, candidate: &SubmissionState) -> Vec<SourceSnippet> {
let recent = self
.store
.list(Some("terminated"), None, self.cfg.similarity_corpus_limit)
.list_champions(self.cfg.similarity_corpus_limit)
.await
.unwrap_or_default();
let mut v = vec![SourceSnippet {
Expand Down
25 changes: 19 additions & 6 deletions crates/prism-challenge/src/score.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,9 @@ pub enum FinalOutcome {
/// The score is **pure bpb**: the LLM review is an anti-cheat / coherence
/// GATE, never a grader — its quality vote and issues are recorded as audit
/// events but never add nor remove points. Agentic `Cheat`/`Suspicious` and
/// cheap similarity `Copied`/`Suspicious` are hard gates (miner-attributable
/// `Score{0}`). Missing agentic verdict is fail-closed upstream as
/// cheap similarity `Copied` are hard gates (miner-attributable `Score{0}`).
/// Cheap LLM `Suspicious` is advisory only (agentic + AST thresholds are the
/// primary judge). Missing agentic verdict is fail-closed upstream as
/// [`FinalOutcome::ChallengeInternal`].
///
/// # Panics
Expand All @@ -53,10 +54,7 @@ pub fn combine_final(outcome: &FinalOutcome) -> prism_store::FinalScore {
if matches!(agentic, VerdictKind::Cheat | VerdictKind::Suspicious) {
return FinalScore::Score(0);
}
if matches!(
similarity,
prism_review::SimilarityKind::Copied | prism_review::SimilarityKind::Suspicious
) {
if matches!(similarity, prism_review::SimilarityKind::Copied) {
return FinalScore::Score(0);
}
FinalScore::Score(score_from_bpb(*bpb))
Expand All @@ -70,6 +68,7 @@ mod final_tests {
use prism_challenge_task::SCORE_MAX;
use prism_review::SimilarityKind::Copied;
use prism_review::SimilarityKind::Original;
use prism_review::SimilarityKind::Suspicious;

#[test]
fn copied_is_hard_zero() {
Expand All @@ -82,6 +81,20 @@ mod final_tests {
assert_eq!(combine_final(&o), prism_store::FinalScore::Score(0));
}

#[test]
fn cheap_suspicious_is_not_hard_zero() {
let o = FinalOutcome::Measured {
bpb: 1.0,
quality: 900,
similarity: Suspicious,
agentic: challenge_agentic::VerdictKind::Clean,
};
assert!(matches!(
combine_final(&o),
prism_store::FinalScore::Score(v) if v > 0
));
}

#[test]
fn agentic_cheat_is_hard_zero() {
let o = FinalOutcome::Measured {
Expand Down
5 changes: 4 additions & 1 deletion crates/prism-challenge/tests/copy_gate.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,9 @@ fn row(
status: Stage,
created_ms: u64,
) -> SubmissionState {
// Terminated priors used as copy-gate victims must be champions (Score>0);
// the gate corpus is tops + ex-tops only.
let final_score = matches!(status, Stage::Terminated).then_some(FinalScore::Score(1));
SubmissionState {
id: id.into(),
miner_hotkey: hotkey.into(),
Expand All @@ -112,7 +115,7 @@ fn row(
arch_id: None,
review: None,
similarity: None,
final_score: None,
final_score,
retry_count: 0,
error_detail: None,
created_at_ms: created_ms,
Expand Down
5 changes: 3 additions & 2 deletions crates/prism-pipeline/src/precheck.rs
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,8 @@ pub fn same_miner(candidate: &SubmissionState, other: &SubmissionState) -> bool
)
}

/// Pre-LLM copy-gate corpus: other miners' prior art only (hotkey + coldkey).
/// Pre-LLM copy-gate corpus: other miners' **champion** prior art only
/// (caller should pass `PrismStore::list_champions`; hotkey + coldkey excluded).
#[must_use]
pub fn gate_corpus_from_rows(
candidate: &SubmissionState,
Expand All @@ -132,7 +133,7 @@ pub fn gate_corpus_from_rows(
.collect()
}

/// Baseline + recent terminated submissions as agentic corpus entries.
/// Baseline + champion submissions as agentic corpus entries.
///
/// Architecture.py only; same-hotkey and same-coldkey prior art excluded.
#[must_use]
Expand Down
56 changes: 56 additions & 0 deletions crates/prism-review/prompts/similarity_v3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
You are the PRISM architecture similarity judge (anti-farming) for a
pretraining-recipe challenge.

The CANDIDATE below is the `architecture.py` of a miner submission. The
CORPUS lists reference architectures: the operator `baseline` and **champion**
architectures only (current top + historical ex-tops with Score>0). Decide
whether the CANDIDATE architecture is effectively a copy of any corpus
architecture.

SCOPE: judge `architecture.py` ONLY. The companion `training.py` is NOT part
of this judgment — the same training script on two different architectures
is legitimate, and the same architecture with a different training script is
still an architecture copy.

Definitions:
- `copied`: near-verbatim architecture, or trivial renaming/formatting
shuffles, or the same model definition with cosmetic deltas (renamed
identifiers, reordered methods, comment edits). Hard zero.
- `suspicious`: strong structural overlap of a *specific* corpus model
(same unique layer stack, same shapes, same forward flow, same custom
blocks) but rewritten enough to blur. Use sparingly — this is advisory
only for operators; do NOT use it for shared modern-LM vocabulary.
- `original`: normal engineering resemblance, standard components, or clear
novelty.

HARD BAN — these are standard modern LM components and MUST NEVER appear as
copy/suspicious evidence by themselves (alone or together):
RMSNorm, LayerNorm, BatchNorm, GroupNorm, Rotary / RoPE / ALiBi / absolute /
relative positional embeddings, SwiGLU / GeGLU / GLU / GELU / SiLU feed-forward,
multi-head / GQA / MQA attention, KV cache, gated residual, parallel residual /
parallel MLP+attention blocks, Pre-Norm / Post-Norm, weight tying, dropout,
FlashAttention, MoE routers, depthwise / causal convolutions used as PE.
Citing any of the above as evidence of copying is a judge error → output
`original` instead.

Only flag `copied` / `suspicious` when the candidate mirrors a *particular*
corpus entry's unique structure (same custom block composition, same unusual
tensor shapes / depths / widths, same novel wiring), not when both use the
same public recipe ingredients.

Shuffling the order of functions/classes does NOT matter.

Output STRICT JSON only:
{"kind": "original|suspicious|copied",
"score": float 0..1,
"closest": "<corpus label or null>",
"evidence": [str, str, str]}
evidence: at most 3 short strings, no markdown. Evidence must name
candidate-specific overlap with a corpus id — never a generic component name
from the ban list above.

=== CANDIDATE architecture.py ===
{ARCH}

=== CORPUS (architecture.py only; champions + baseline) ===
{CORPUS}
Loading
Loading