From 8d3cc7e225f6ffbe46526f5e94467c5d33a82b89 Mon Sep 17 00:00:00 2001 From: echobt <154886644+echobt@users.noreply.github.com> Date: Tue, 11 Aug 2026 16:23:36 +0000 Subject: [PATCH] docs: sync AutoModel recipe 2.0 miner guide Update public miner docs for live recipe 2.0 (automodel@v0.5.0 pin + patch, BYOK X-Lium-Api-Key) and retire the architecture.py/training.py path. --- README.md | 58 ++++---- docs/README.md | 21 +-- docs/api.md | 23 +-- docs/getting-started.md | 120 ++++++++-------- docs/prism.md | 276 ++++++++++++++++++++++++++++++++++++ docs/scoring.md | 55 +++---- docs/submit.md | 119 +++++++--------- docs/troubleshooting.md | 41 +++--- examples/baseline/README.md | 68 +++------ 9 files changed, 512 insertions(+), 269 deletions(-) create mode 100644 docs/prism.md diff --git a/README.md b/README.md index c8b0c4f..086cb5e 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # PRISM -**Miner guide for the BASE prism challenge — HTTP recipe submit.** +**Miner guide for the BASE prism challenge — HTTP AutoModel patch submit.** [![BASE](https://img.shields.io/badge/BASE-subnet-black.svg)](https://github.com/BaseIntelligence/base) [![Bittensor](https://img.shields.io/badge/Bittensor-subnet-black.svg)](https://bittensor.com/) @@ -15,18 +15,19 @@ [Submit](docs/submit.md) · [Scoring & competition](docs/scoring.md) · [API](docs/api.md) · -[Examples](examples/baseline/) +[Full guide](docs/prism.md) ## What it is -PRISM is a research challenge: you try **new architectures** and the challenge re-executes -them fairly. You submit **two Python scripts** — `architecture.py` (`build_model(ctx)`) and -`training.py` (`train(model, ctx)`) — and the operator runs them on a GPU pod against a -pinned FineWeb-Edu shard. Score is pure **bits-per-byte** (bpb, lower is better) measured -by the operator harness. There is **no** miner Docker image, no CVM, no on-chain write from -miners — HTTP submit only. +PRISM is a research challenge on a pinned +[NeMo AutoModel](https://github.com/NVIDIA-NeMo/Automodel) base: you fork the +operator pin, edit under that tree, and submit a **unified git diff**. The +operator applies your patch fail-closed, then re-executes training on a +miner-funded Lium GPU pod against a pinned FineWeb-Edu shard. Score is pure +**bits-per-byte** (bpb, lower is better). There is **no** miner Docker image, +no CVM, no on-chain write from miners — HTTP submit only. | | | |---|---| @@ -34,40 +35,43 @@ miners — HTTP submit only. | Production gateway | `https://chain.joinbase.ai` | | Staging gateway | `http://staging.api.joinbase.ai` | | Submit path | `/challenge/prism/v1/submissions` | -| Recipe | v1.2.0 — telemetry hooks required | +| Recipe | **2.0.0** — AutoModel pin + patch (`automodel@v0.5.0`) | +| Live GPU | Miner-funded Lium — pass `X-Lium-Api-Key` | -This repository holds **miner documentation and examples only**. Control-plane source -lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). +This repository holds **miner documentation and examples only**. Control-plane +source lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). ## Start here -1. Read [Getting started](docs/getting-started.md). -2. Copy [`examples/baseline/`](examples/baseline/) — it shows the required telemetry - hooks (`prism_telemetry.report` + `finish_evaluation`). -3. Zip `architecture.py` + `training.py` and submit — see [Submit](docs/submit.md). -4. Poll events until `terminated`, then check your bpb — see [API](docs/api.md). +1. Read [Getting started](docs/getting-started.md) (or the [full guide](docs/prism.md)). +2. `GET /v1/recipe` — copy `automodel_pin_id`, `automodel_git_commit`, and caps. +3. Checkout that AutoModel commit → edit → `git diff > automodel.patch`. +4. Pack `automodel.base` + `automodel.patch` (+ optional `prism.toml`) and submit + with your hotkey + **`X-Lium-Api-Key`** — see [Submit](docs/submit.md). +5. Poll events until `terminated`, then check your bpb — see [API](docs/api.md). ```bash export GATEWAY=https://chain.joinbase.ai export HOTKEY=<64 lowercase hex> # public hotkey only — never a secret key +export LIUM_API_KEY= -cd examples/baseline -zip -j submission.zip architecture.py training.py +# After forking the pin and producing automodel.base + automodel.patch: +zip -j submission.zip automodel.base automodel.patch # + prism.toml if used curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ -H 'content-type: application/zip' \ -H "X-Miner-Hotkey: $HOTKEY" \ + -H "X-Lium-Api-Key: $LIUM_API_KEY" \ --data-binary @submission.zip ``` ## The three things miners get wrong -1. **Missing telemetry hooks** — `training.py` must import `prism_telemetry` and call - `report(...)` during training (and may call `finish_evaluation()` to stop early). - Missing hooks = hard contract violation, zero score, terminal. -2. **Copying someone's `architecture.py`** — the pre-GPU copy gate rejects byte/AST - copies of *earlier* architectures with zero score, no appeal. Starting from the - published baseline is fine. -3. **Submitting again while gated** — one accepted architecture submission per hotkey; - a second one returns `409 submission_gated`. Training-only entries on published - architectures are separate slots (one per `(hotkey, arch_id)`). +1. **Legacy 1.x ZIPs** — `architecture.py` + `training.py` (or training-only + `arch_id`) return `400 unsupported_layout` / `recipe_version` on live 2.0. + Ship `automodel.base` + `automodel.patch` only. +2. **Wrong pin / stale diff** — `automodel.base` must equal live + `automodel_pin_id` (`automodel@v0.5.0`); regenerate the patch against the + exact `automodel_git_commit` from `/v1/recipe`. +3. **Missing `X-Lium-Api-Key`** — live eval runs on **your** Lium account. + Missing key → `400 missing_lium_api_key`. diff --git a/docs/README.md b/docs/README.md index 2dfa34a..21aa4d6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,13 +1,18 @@ # PRISM miner docs +Live recipe is **2.0.0**: submit an AutoModel pin id + unified git diff +(`automodel.base` + `automodel.patch`), not a free-form two-script ZIP. + | Page | What it covers | |------|----------------| -| [Getting started](getting-started.md) | Contract, hooks, dataset, budgets | -| [Submit](submit.md) | ZIP/JSON submit, gating, retries, training-only entries | -| [Scoring & competition](scoring.md) | bpb lattice, anti-copy, architecture competition, top-model | -| [API](api.md) | Routes, statuses, telemetry | -| [Troubleshooting](troubleshooting.md) | Common failures | -| [`examples/baseline/`](../examples/baseline/) | Reference recipe with hooks | +| [Getting started](getting-started.md) | Fork pin → edit → `git diff` → pack ZIP | +| [Submit](submit.md) | ZIP/JSON, BYOK Lium key, gating, precheck, retries | +| [Scoring & competition](scoring.md) | bpb lattice, patch anti-copy, causal ban, top-model | +| [API](api.md) | Routes, statuses, diff + telemetry | +| [Troubleshooting](troubleshooting.md) | `unsupported_layout`, pin/patch failures, Lium | +| [Full guide](prism.md) | Complete miner guide (mirrors BASE `docs/external-miner/prism.md`) | -Normative sources (BASE monorepo): `docs/PRISM.md`, `docs/PRISM_RECIPE.md`, -`docs/external-miner/prism.md`. +Normative sources (BASE monorepo): +[`docs/PRISM.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM.md), +[`docs/PRISM_RECIPE.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM_RECIPE.md), +[`docs/external-miner/prism.md`](https://github.com/BaseIntelligence/base/blob/main/docs/external-miner/prism.md). diff --git a/docs/api.md b/docs/api.md index 7b6ee2b..311ade7 100644 --- a/docs/api.md +++ b/docs/api.md @@ -9,16 +9,16 @@ Replace `{GATEWAY}` with `https://chain.joinbase.ai` (prod) or | Route | What it tells you | |-------|-------------------| -| `POST /challenge/prism/v1/submissions` | Submit zip / JSON / training-only | -| `POST /challenge/prism/v1/submissions/precheck` | Advisory copy-gate (3/coldkey/UTC day; no queue/pod) | +| `POST /challenge/prism/v1/submissions` | Submit AutoModel ZIP / JSON (`automodel.base` + `automodel.patch`) | +| `POST /challenge/prism/v1/submissions/precheck` | Advisory copy/layout gate (3/coldkey/UTC day; no queue/pod) | | `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic records | +| `GET /challenge/prism/v1/submissions/{id}/diff` | Unified diff + diffstat / classification (recipe ≥ 2.0) | | `GET /challenge/prism/v1/submissions/{id}/events` | Stage timeline | -| `POST /challenge/prism/v1/submissions/{id}/retry` | Requeue an infra-failed row | +| `POST /challenge/prism/v1/submissions/{id}/retry` | Requeue an infra-failed row (within recovery window) | | `GET /challenge/prism/v1/submissions?miner=` | Your submissions | -| `GET /challenge/prism/v1/architectures` | Published archs + per-arch best bpb | -| `GET /challenge/prism/v1/recipe` | Versioned recipe descriptor + pin | -| `GET /challenge/prism/v1/recipe/baseline` | Official baseline scripts | +| `GET /challenge/prism/v1/recipe` | Caps + AutoModel pin (`automodel_pin_id`, commit, content sha) | | `GET /challenge/prism/v1/status` | Backend / epoch / queues / recipe pin | +| `GET /challenge/prism/v1/jobs` | Active/recent pods (ops visibility) | | `GET /v1/site/arenas/prism/submissions/{id}/telemetry` | Loss curve / gradients / layer stats | ## Poll example @@ -29,6 +29,7 @@ SUB= curl -sS "$GATEWAY/challenge/prism/v1/submissions/$SUB" curl -sS "$GATEWAY/challenge/prism/v1/submissions/$SUB/events" +curl -sS "$GATEWAY/challenge/prism/v1/submissions/$SUB/diff" ``` ## Status values @@ -40,12 +41,13 @@ Terminal states to know: | Status | Meaning | |--------|---------| -| `rejected` | Pre-LLM copy gate: byte/AST copy of an *earlier* architecture (`Score(0)`, no GPU time, no LLM review) | +| `rejected` | Copy / layout / causal gate: terminal `Score(0)` (often before GPU) | | `failed` | Infra retries exhausted (`auto_retry` events) or harness/internal failure | | `terminated` with `score.kind = "no_score"` | `ChallengeInternal` — operator-side, never a miner zero | | `terminated` with score 0 | Cheat / suspicious / copied verdict (see the `scoring` event detail) | -Submit errors: `403 hotkey_not_in_metagraph`, `404 unknown_arch`, +Submit errors: `400 unsupported_layout`, `400 recipe_version`, +`400 missing_lium_api_key`, `403 hotkey_not_in_metagraph`, `409 submission_gated`, `503 metagraph_unavailable`. Precheck errors: same membership/contract codes, plus @@ -53,8 +55,9 @@ Precheck errors: same membership/contract codes, plus ## Auth note -Miner routes identify you by hotkey (`X-Miner-Hotkey` or JSON `miner_hotkey`). Never -send challenge signing keys or gateway owner keys from a miner client. +Miner routes identify you by hotkey (`X-Miner-Hotkey` or JSON `miner_hotkey`). +On live, also send **`X-Lium-Api-Key`** (your funded Lium account). Never send +challenge signing keys or gateway owner keys from a miner client. ## Next diff --git a/docs/getting-started.md b/docs/getting-started.md index f7b25c2..a393eea 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,54 +1,57 @@ # Getting started -## The contract (recipe v1.2.0) +## The contract (recipe v2.0.0) -You ship **two scripts only**. The operator harness (`prism_harness.py`) imports them, -downloads the pinned dataset, verifies its SHA-256, times the run, and reports -`METRICS_JSON` (bpb, tokens, steps, wall clock, gpu, params). +You do **not** ship a free-form `architecture.py` / `training.py` project. +Live recipe **2.0.0** accepts only a pin id plus your unified diff against that +pin: -```python -# architecture.py -def build_model(ctx): - """Return a model given the recipe context (devices, dims, seed).""" - -# training.py -def train(model, ctx): - """Train the model; must respect ctx.budget(): - budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train).""" +```text +automodel.base # required — pin id from GET /v1/recipe (live: automodel@v0.5.0) +automodel.patch # required — unified diff vs that pin (git diff pin...HEAD) +prism.toml # optional — entry / model-config knobs ``` -No third source file, no offline weights, no network at pod runtime beyond the pinned -dataset pull. +**Workflow: fork pin → edit → `git diff` → submit** -## Telemetry hooks (required since recipe 1.1.0) +1. Read the live pin from `GET /v1/recipe` (`automodel_pin_id`, + `automodel_repo_url`, `automodel_git_commit`, `automodel_content_sha256`). +2. Check out that exact AutoModel commit (or extract the staged archive and + verify `automodel_content_sha256` matches `/v1/recipe`). +3. Edit under the AutoModel layout — new model modules / configs are allowed; + trainer / data-path edits get high scrutiny. +4. Produce a unified diff against the pin commit, e.g. + `git diff > automodel.patch`. +5. Write `automodel.base` as a single line equal to `automodel_pin_id`, pack + the ZIP, and `POST /v1/submissions` with your hotkey + **`X-Lium-Api-Key`**. -The harness registers a `prism_telemetry` module before your code loads (also at -`ctx["telemetry"]`). Your `training.py` **MUST**: +Models must stay **≤ 350M parameters**. The pod has **no network** +(`unshare --net`) beyond the operator-owned dataset pull — do not call Hub +downloads from miner code. -```python -import prism_telemetry +**Legacy recipe 1.x is rejected on live.** Two-script ZIPs +(`architecture.py` + `training.py`), 1.3 source-tree ZIPs, and training-only +`arch_id` submissions return `400 unsupported_layout` or `400 recipe_version`. +Do not ship Megatron-Bridge or other non-AutoModel frameworks. -prism_telemetry.report(loss=..., step=..., grad_norm=..., layer_stats=...) # every N steps -prism_telemetry.finish_evaluation() # optional early stop: score the model as-is -``` +## Pay for your own GPU (required on live) -- `report(...)` feeds the loss/gradient/layer series persisted master-side and shown on - the site (`/v1/site/arenas/prism/submissions/{id}/telemetry`). -- `finish_evaluation()` raises a `BaseException` through `train()` so your own - `except Exception` blocks cannot swallow it; without it the eval ends when `train()` - returns or the wall-clock cap fires. -- **Missing hooks are a hard contract violation**: the review fails the submission - (`missing_telemetry_hooks`, zero score, terminal — no retry). +Create a [Lium](https://lium.io) account, fund it, and pass your API key on +every live submit: + +```http +X-Lium-Api-Key: +``` -The [baseline example](../examples/baseline/) shows the exact pattern, including an -offline fallback stub for local testing. +The key is held only in master memory for that submission (never stored in the +DB, never logged). Missing key on live → `400 missing_lium_api_key`. -## Training-only submissions (architecture competition, recipe ≥ 1.2.0) +## Telemetry hooks (still required) -Instead of shipping both scripts you can submit `training.py` + `arch_id` referencing a -**published** architecture. The master pulls `architecture.py` from the registry; the -same harness contract applies unchanged. Published archs: `GET /v1/architectures`. -See [Submit](submit.md#training-only-entries). +The harness wrap still requires `prism_telemetry` reporting / +`finish_evaluation` under the AutoModel train entry. Patches that remove or +bypass those hooks fail review (`missing_telemetry_hooks`, zero score, +terminal). ## Pinned dataset @@ -60,43 +63,32 @@ See [Submit](submit.md#training-only-entries). | SHA-256 | `e5a2eae25f057f0856a10bfae314c6ca8ea8bb08456d2131e9e89b2b8305e2f6` | The harness re-verifies the hash on the file it actually fetched; a mismatch ends the -eval as `ChallengeInternal` — never a miner score. +eval as `ChallengeInternal` — never a miner score. Always confirm live values via +`GET /v1/recipe`. ## Budget & caps | Cap | Value | |-----|-------| -| Train wall clock | 6.0 h per submission | -| Pod lifetime | 7.0 h (train + bootstrap margin) | -| Hard step cap | 20 000 | -| Source size | 128 KiB per script | -| Model parameters | ≤ **350 000 000** after `build_model` | -| `train_rows` (from `GET /v1/recipe`) | **2048** — baseline / default cut in `ctx` | -| `val_rows` | **256** — frozen val scored by the harness | - -`train_rows` is what the **sealed baseline** trains on (~2M GPT-2 tokens for -that slice). It is **not** a hard “you only get 2048 rows” ceiling for -competitive recipes: the harness gives you the full pinned parquet at -`ctx["dataset_path"]`, and you may stream it until the 6h / 20k-step guard -fires. Token count then depends on your loop and the GPU — a long Lium run can -reach ~O(10⁹) tokens. Marketing charts that once said “2.6B tokens · single -pass” were showing a leader’s **observed** telemetry, not a fixed recipe -quota. Always trust live `GET /v1/recipe` (`pin_hex`, `train_rows`, caps). - -The sealed baseline is deliberately mediocre (short cut, few steps). Matching -a board BPB near ~4–5 requires a competitive trainer, not an unmodified -baseline on a 4090 for a few minutes. +| Train wall clock | 6.0 h per submission (`train_hours_cap`) | +| Hard step cap | 20 000 (`max_train_steps`) | +| Model parameters | ≤ **350 000 000** (`max_params`) | + +Trust live `GET /v1/recipe` (`version`, `automodel_*`, `pin_hex`, caps) over any +marketing chart. ## Recipe pin -`GET /v1/recipe` returns the versioned descriptor (dataset URL/hash, caps, -`train_rows` / `val_rows`, recipe version, `pin_hex`). Production today is -recipe **1.2.0** — open docs PRs that advertise 1.3+/1.4.0/v3 scoring describe -**unreleased** control-plane work (`prism-better`), not what -`https://chain.joinbase.ai` executes. `GET /v1/recipe/baseline` returns the -official baseline scripts — the best starting point for your own architecture. +```bash +curl -sS "$GATEWAY/challenge/prism/v1/recipe" +``` + +Live recipe **2.0.0** advertises `version: "2.0.0"` and AutoModel pin fields +(`automodel_pin_id` = `automodel@v0.5.0`, `automodel_repo_url`, +`automodel_git_ref`, `automodel_git_commit`, `automodel_content_sha256`). ## Next → [Submit](submit.md) +→ [Full guide](prism.md) → [Scoring & competition](scoring.md) diff --git a/docs/prism.md b/docs/prism.md new file mode 100644 index 0000000..b8254fd --- /dev/null +++ b/docs/prism.md @@ -0,0 +1,276 @@ + + +# Prism challenge — HTTP AutoModel patch submit + +**challenge_id:** `prism` +**scoring_version:** `2` live (bpb-only; LLM review is an anti-cheat gate, not a grader). **v3 (opt-in, shadow-by-default):** composite scoring runs alongside — your run is also measured on the G1–G8 battery; see *v3 scoring* below. +**recipe_version:** `2.0.0` (pinned [NeMo AutoModel](https://github.com/NVIDIA-NeMo/Automodel) base + miner unified diff; legacy 1.x layouts rejected on live) +**Path:** HTTP only — **no Phala/CVM** + +Normative docs (BASE monorepo): +[`PRISM.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM.md), +[`PRISM_RECIPE.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM_RECIPE.md). + +## What you submit + +A **ZIP** (preferred) — or JSON with the same members / `zip_base64` — that is +**not** a free-form `architecture.py` / `training.py` project. Recipe **2.0.0** +accepts only an AutoModel pin id plus your git diff against that pin: + +```text +automodel.base # required — pin id from GET /v1/recipe (live: automodel@v0.5.0) +automodel.patch # required — unified diff vs that pin (git diff pin...HEAD) +prism.toml # optional — entry / model-config knobs +``` + +**Workflow: fork pin → edit → `git diff` → submit** + +1. Read the live pin from `GET /v1/recipe` (`automodel_pin_id`, + `automodel_repo_url`, `automodel_git_commit`, `automodel_content_sha256`). +2. Check out that exact AutoModel commit (or extract the staged archive and + verify `automodel_content_sha256` matches `/v1/recipe`). +3. Edit under the AutoModel layout — new model modules / configs are allowed; + trainer / data-path edits get high scrutiny. +4. Produce a unified diff against the pin commit, e.g. + `git diff > automodel.patch`. +5. Write `automodel.base` as a single line equal to `automodel_pin_id`, pack + the ZIP, and `POST /v1/submissions` with your hotkey + **`X-Lium-Api-Key`**. + +Models must stay **≤ 350M parameters**. The pod has **no network** +(`unshare --net`) beyond the operator-owned dataset pull — do not call Hub +downloads from miner code. + +**Legacy recipe 1.x rejected on live.** Two-script ZIPs +(`architecture.py` + `training.py`), 1.3 source-tree ZIPs, and training-only +`arch_id` submissions return `400 unsupported_layout` or `400 recipe_version` +once 2.0 is advertised. Do not ship Megatron-Bridge or other non-AutoModel +frameworks. + +**Telemetry.** The harness wrap still requires `prism_telemetry` reporting / +`finish_evaluation` under the AutoModel train entry. Patches that remove or +bypass those hooks fail review (`missing_telemetry_hooks`, zero score, +terminal). + +**Diff visibility.** After intake, inspect your applied delta at +`GET /v1/submissions/{id}/diff` (full unified diff + diffstat / classification). + +Evaluation runs on **miner-funded** Lium GPU pods (you pay the rent). Master +still operates the pod over SSH; you do **not** deploy a miner CVM. CI uses +`SimLiumBackend` and does not need a key. + +## Pay for your own GPU (required on live) + +Create a [Lium](https://lium.io) account, fund it, and pass your API key on +every live submit: + +```http +X-Lium-Api-Key: +``` + +The key is held only in master memory for that submission (never stored in the +DB, never logged). Missing key on live → `400 missing_lium_api_key`. Cost +guardrails (`max_price_per_hour`, lifetime) still apply so a bad key cannot +rent unbounded SKUs through the orchestrator. + +## Submit + +```bash +# ZIP via gateway (preferred) +curl -sS -X POST "$BASE_GATEWAY/challenge/prism/v1/submissions" \ + -H 'content-type: application/zip' \ + -H "X-Miner-Hotkey: <64 lowercase hex>" \ + -H "X-Lium-Api-Key: $LIUM_API_KEY" \ + --data-binary @submission.zip + +# JSON sources (local/CI convenience) +curl -sS -X POST "$BASE_GATEWAY/challenge/prism/v1/submissions" \ + -H 'content-type: application/json' \ + -H "X-Lium-Api-Key: $LIUM_API_KEY" \ + -d @submission.json + +# Local / direct +curl -sS -X POST "http://127.0.0.1:28092/v1/submissions" \ + -H 'content-type: application/json' \ + -H "X-Lium-Api-Key: $LIUM_API_KEY" \ + -d @submission.json +``` + +Inspect recipe + AutoModel pin before coding: + +```bash +curl -sS "$BASE_GATEWAY/challenge/prism/v1/recipe" +``` + +Live recipe **2.0.0** advertises `version: "2.0.0"` and AutoModel pin fields +(`automodel_pin_id` = `automodel@v0.5.0`, `automodel_repo_url`, +`automodel_git_ref`, `automodel_git_commit`, `automodel_content_sha256`), +plus caps such as `train_hours_cap: 6.0`, `max_train_steps: 20000`, +`max_params: 350000000`, FineWeb dataset pin, and `pin_hex` (sha over the +versioned descriptor). Trust `/v1/recipe`, not marketing chart labels. + +`POST /v1/submissions` is idempotent by `submission_id` (hash of **pin id ‖ +`0x00` ‖ patch bytes**). + +## Submission gating (1-max) + +- Your hotkey must be **registered on the subnet** (metagraph). Unknown hotkey + → `403 hotkey_not_in_metagraph`; a fresh registration may lag the snapshot + (`503 metagraph_unavailable` → retry shortly). +- **One accepted patch submission per hotkey.** While yours is `registered` / + `rejected`, or `blocked` **outside** the infra recovery window, a *different* + patch submission gets + `409 submission_gated`. Re-POSTing the **identical** pin+patch is always + safe (idempotent `200 already-queued`). +- If your hotkey **leaves the metagraph**, the watcher reopens your slot(s) + automatically — resubmit under your new uid. +- Infra failures (Lium pod, review/similarity/LLM infra) **auto-retry up to 3 + times**; cheat / rejected verdicts are terminal. After an infra failure + (`ChallengeInternal`), you may **resubmit within 30 minutes** (new POST or + `POST /v1/submissions/{id}/retry`). After 30 minutes the slot stays blocked + until your hotkey leaves the metagraph. + +## Anti-copy rule (patch / delta) + +Copying another miner's **patch** (or an equivalent touched-file rewrite of +an earlier champion delta) is terminal `rejected` with zero score — judged +before or without burning GPU when the gate can decide from the diff alone. +Review focuses on your unified diff and touched files (`arch` / `trainer` / +`data` / `other`), not the whole AutoModel tree. Starting from the operator +pin and submitting only your delta is the intended path. + +## Causal LM contract (banned: non-causal label leak) + +Prism scores **next-token** cross-entropy → BPB. Architectures must not let +position `t` read tokens `t+1…` (including the label). Dense sequence mixers — +MLP-Mixer-style `TokenMix` / `t_mix` / `nn.Linear` over the full time axis +after `transpose(1, 2)` — **without** a causal mask (`triu` / `tril` / +`is_causal` / attention mask) are a hard ban (`non_causal_label_leak`, +`Score(0)`, terminal, often caught **before** GPU rent). Channel mixing and +causal attention / causal conv are fine; bidirectional full-sequence mixes +used as a next-token LM are not. + +### Precheck before you submit (recommended) + +Dry-run the copy / layout gate **without** burning your 1-max slot or a +GPU eval (send the same AutoModel ZIP you would submit): + +```bash +curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions/precheck" \ + -H 'content-type: application/zip' \ + -H "X-Miner-Hotkey: $HOTKEY" \ + --data-binary @submission.zip +``` + +| Field | Meaning | +|-------|---------| +| `similar` | `true` → would hard-reject at intake copy gate | +| `verdict` | `clean` / `copied` / `skipped` | +| `matched_against` | Corpus id only (never competitor source) | +| `score` | Similarity in `[0,1]` when compared | +| `quota` | `{ day, used, limit: 3, remaining, identity }` | + +**Quota: 3 attempts per coldkey per UTC day** (falls back to hotkey when the +metagraph Owner coldkey is unknown). Rotating hotkeys under the same coldkey +does **not** reset the budget. A 4th call returns `429` / +`precheck_quota_exceeded` with `remaining=0`. Precheck never creates a scored +submission and never rents a Lium pod. + +## Scoring (summary) + +Final leaf score is pure bits-per-byte (bpb) on the lattice `[0, SCORE_MAX]`. +The shared **agentic** gate (AST + metrics/receipt) hard-zeros `cheat` / +`suspicious`. Cheap LLM similarity hard-zeros `Copied`, and `Suspicious` only +when confidence `≥ 0.9` with non-generic evidence (below that — e.g. 0.7 citing +RMSNorm/SwiGLU/LayerNorm — does **not** wipe your score). Copy/similarity +corpora are **champions only** (current top + historical Score>0 ex-tops) plus +baseline — not every past submission — and still exclude your own prior art +(same hotkey **or** same coldkey). Standard components (RMSNorm, RoPE, SwiGLU, +LayerNorm, gated/parallel residual, …) are **not** plagiarism signals. LLM +quality is coherence-only, not a grader. +Public gallery/leaderboard show champions only. +**Competition (temporary):** emission uses **your own best training score +only** — architecture-owner credit (rewarding arch owners when others train +well on their code) is **disabled** for now so the best-BPB trainer keeps +Prism's weights. Emission remains **winner-take-all**: only the single highest +own score that epoch receives Prism's share (50% of the subnet); ties break by +lexicographically smallest hotkey. Scores first land in the leaf +set emitted at the first chain-epoch boundary **after** your run finalizes (a +long train that crosses epochs is normal — outbox assignment is exactly once). +Positive scores then keep participating in later epochs' competition sets until +a better valid score supersedes them (WTA still collapses to one leaf winner). +The global-best model (sources + `ARTIFACT.json` / checkpoint release) is +published to +[`BaseIntelligence/prism`](https://github.com/BaseIntelligence/prism) +`top-model/`. + +## v3 scoring (shadow-by-default) + +Recipe ≥ 1.3.0 harnesses run a **two-phase pod flow**: your code trains +(`phase=train`), checkpoints, and only then does the operator stage private +eval assets — the eval phase (`phase=eval`) is a fresh subprocess that runs +the frozen-val bpb plus the **G1–G8 battery**: intrinsic fit (G1), +commonsense/reading (G2), retrieval/recall (G3), reasoning (G4), +long-context (G5), sample efficiency from the train probe curve (G6), +inference efficiency (G7), and training stability/µP (G8). Everything the +battery reports is organizer-measured (**Zone A**, `org.*`) and is computed +inside the harness — your code never emits it. + +**G5 is pretrain-only (recipe ≥ 1.4.0).** The long-context group scores a +**base LM**, not an instruction-tuned chat model: completion-style / +few-shot base prompts, short exact-match or multiple-choice logprob — +no chat templates, no free-form summarization, no LLM-as-judge on the +ranked path. Length targets are counted in tokens of **your** tokenizer +(`ctx["tokenizer"]`). Scored keys (group weight 0.15 total): +`org.g5.ruler_acc` (0.35), `org.g5.babilong_acc` (0.25), +`org.g5.natural_mcq_acc` (0.15), `org.g5.helmet_rag_acc` (0.15), +`org.g5.lstar` (0.10). L* is the highest length where pooled +RULER+BABILong accuracy stays ≥ 90% of the shortest-grid accuracy and +≥ 0.25 (else 0). Natural MCQ / HELMET RAG packs are mirrored like G2/G4. + +Your `train()` return dict (`train_metrics` in METRICS_JSON v2) is +**Zone B**: participant-reported, displayed-but-labelled, validated at +ingest (scalars/series/histograms under `miner..`, caps +64 scalars / 16 series / 10k points / 1 MB), and **never scored**. Do not +emit `org.*` keys — that quarantines the report as anti-cheat evidence. +You can also post additional self-reports out-of-band: +`POST /v1/submissions/{id}/zone-b` with a JSON envelope +`{"schema_version": "", "prev_hash": , "metrics": {"miner..": {"kind": "scalar"|"series"| +"histogram", ...}}}`. Reports chain per submission (`prev_hash` → previous +`report_hash`; omit it for master-chained ingest), are validated against +organizer ground truth (token/step/wall-clock counters, MFU ceiling, +terminal-loss band) and the cross-miner cohort, and land a stored verdict +(`ok` / `flagged` / `quarantined`) — verdicts are evidence, never an +auto-zero. Malformed or over-cap envelopes reject `422` and store nothing. + +While `PRISM_SCORING_MODE=shadow` (default) the leaf score stays pure bpb, +bit-identical to v2. After the reference baselines are measured and the +anchor set is pre-registered, governance may flip to `composite`: group +scores are anchor-normalized, gate-filtered (`g3 ≥ 0.25`, `g8 ≥ 0.5`, +budget + CI gates), combined as a weighted geometric mean, and ranked by +the bootstrap lower-confidence bound +(`lattice = round(SCORE_MAX × max(0, C − 1.645·SE))`). Inspect the anchor +registry and pre-registration commits at `GET /v1/anchors` and +`GET /v1/preregistration`; per-run Zone A / Zone B rows at +`GET /v1/submissions/{id}/metrics?zone=a|b`. + +## Useful routes + +| Route | Use | +|-------|-----| +| `POST /v1/submissions/precheck` | Advisory copy/layout gate (3/coldkey/UTC day); no submit | +| `GET /v1/status` | Backend mode, epoch, queue | +| `GET /v1/recipe` | Caps + AutoModel pin (`automodel_pin_id`, commit, content sha) | +| `GET /v1/submissions/{id}` | Detail + receipt + scores + composite block (v3) | +| `GET /v1/submissions/{id}/diff` | Unified diff + diffstat / classification (recipe ≥ 2.0) | +| `GET /v1/submissions/{id}/events` | Stage timeline | +| `GET /v1/submissions/{id}/metrics?zone=a\|b` | Zone A battery rows / Zone B self-report chain (v3) | +| `POST /v1/submissions/{id}/zone-b` | Miner Zone B self-report intake: validated + chained + stored (v3) | +| `GET /v1/anchors` | v3 anchor-set registry + status | +| `GET /v1/preregistration` | v3 anchor pre-registration hash-commits | +| `GET /v1/site/arenas/prism/submissions/{id}/telemetry` | Miner-reported loss curve / gradients / layer stats (from `prism_telemetry.report`) | +| `GET /v1/jobs` | Active/recent pods (ops) | +| `GET /health` | Liveness | + +Emission share for prism is owner-controlled via the trust root. Current split is +`5000` bps prism / `5000` bps design (50/50). diff --git a/docs/scoring.md b/docs/scoring.md index 6b0c6c0..70f6757 100644 --- a/docs/scoring.md +++ b/docs/scoring.md @@ -6,20 +6,22 @@ lower bpb, higher score. The LLM reviews are **gates, not graders**: they verify the submission is coherent and not cheating; their quality notes never move the score. -## Anti-copy (architecture-only) +## Anti-copy (patch / delta) -- A **pre-LLM copy gate** compares your `architecture.py` against earlier submissions - (byte hash + AST fingerprints, `created_at` ordered). A byte/AST copy of a - strictly-earlier architecture is terminal `rejected` with zero score — **no GPU time, - no LLM review**, no appeal. The published baseline is exempt. -- Similarity is judged on `architecture.py` **only**: `training.py` is exempt on both - sides — the same training loop on two different architectures is legitimate, and - training-only entries on a published arch are never "copies" by construction. -- After the gate, an LLM similarity review + agentic anti-cheat still run: - `Copied` / `Suspicious` / `cheat` verdicts → hard zero. -- Probe the cheap gate first with `POST /v1/submissions/precheck` (3/coldkey/UTC - day) — see [Submit](submit.md#precheck-similarity-before-you-submit). +Copying another miner's **patch** (or an equivalent touched-file rewrite of +an earlier champion delta) is terminal `rejected` with zero score — judged +before or without burning GPU when the gate can decide from the diff alone. +Review focuses on your unified diff and touched files (`arch` / `trainer` / +`data` / `other`), not the whole AutoModel tree. Starting from the operator +pin and submitting only your delta is the intended path. +After the gate, an LLM similarity review + agentic anti-cheat still run: +`Copied` / high-confidence `Suspicious` (≥ 0.9 with non-generic evidence) / +`cheat` → hard zero. Standard components (RMSNorm, RoPE, SwiGLU, LayerNorm, +…) are **not** plagiarism signals. + +Probe the cheap gate first with `POST /v1/submissions/precheck` (3/coldkey/UTC +day) — see [Submit](submit.md#precheck-similarity-before-you-submit). ## Causal LM contract (banned: non-causal label leak) @@ -31,27 +33,28 @@ attention mask) are a hard ban (`non_causal_label_leak`, `Score(0)`, terminal, often caught **before** GPU rent). Channel mixing and causal attention / causal conv are fine; bidirectional full-sequence mixes used as a next-token LM are not. -## Architecture competition (emission math) - -Per epoch, your emission is the **max** of: - -1. **Challenger credit** — your own best training result this epoch (any arch), and -2. **Owner credit** — for each architecture you own, that arch's **best result by any - trainer** this epoch. +## Competition (emission) -Max, never summed — architecture owners are rewarded when *anyone* trains well on -their architecture. `Score(0)` rows (cheat / copy-gate) never set an arch's best. +**Competition (temporary):** emission uses **your own best training score +only** — architecture-owner credit (rewarding arch owners when others train +well on their code) is **disabled** for now so the best-BPB trainer keeps +Prism's weights. Emission remains **winner-take-all**: only the single highest +own score that epoch receives Prism's share (50% of the subnet); ties break by +lexicographically smallest hotkey. -Published architectures and their best bpb so far: `GET /v1/architectures`. +Scores first land in the leaf set emitted at the first chain-epoch boundary +**after** your run finalizes. Positive scores then keep participating in later +epochs' competition sets until a better valid score supersedes them (WTA still +collapses to one leaf winner). ## Top-model publish Whenever a new **global-best bpb** lands, the master publishes the winning -`architecture.py` + `training.py` + `METRICS.json` to +sources + `ARTIFACT.json` / checkpoint release to [`BaseIntelligence/prism`](https://github.com/BaseIntelligence/prism) under -[`top-model/`](https://github.com/BaseIntelligence/prism/tree/main/top-model) and -journals the publication. The `top-model/` directory always mirrors the current -champion; history lives in git. +[`top-model/`](https://github.com/BaseIntelligence/prism/tree/main/top-model) +and journals the publication. The `top-model/` directory always mirrors the +current champion; history lives in git. ## Telemetry diff --git a/docs/submit.md b/docs/submit.md index b2d012e..d539cc9 100644 --- a/docs/submit.md +++ b/docs/submit.md @@ -1,6 +1,7 @@ # Submit -Preferred path: a **ZIP** through the production or staging gateway. +Preferred path: a **ZIP** through the production or staging gateway with +`automodel.base` + `automodel.patch` (+ optional `prism.toml`). ## ZIP (preferred) @@ -9,41 +10,39 @@ Preferred path: a **ZIP** through the production or staging gateway. | Method / URL | `POST {GATEWAY}/challenge/prism/v1/submissions` | | `Content-Type` | `application/zip` | | `X-Miner-Hotkey` | 64 lowercase hex | -| `X-Prism-Arch-Id` | training-only entries: published `arch_id` (zip then contains `training.py` only) | -| Body | raw zip bytes (`architecture.py` + `training.py` at the root) | +| `X-Lium-Api-Key` | your Lium API key (required on live) | +| Body | raw zip bytes (`automodel.base` + `automodel.patch` at the root) | ```bash export GATEWAY=https://chain.joinbase.ai export HOTKEY=<64 lowercase hex> +export LIUM_API_KEY= -cd examples/baseline -zip -j submission.zip architecture.py training.py +# automodel.base = single line equal to recipe automodel_pin_id (automodel@v0.5.0) +# automodel.patch = git diff +zip -j submission.zip automodel.base automodel.patch # + prism.toml if used curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ -H 'content-type: application/zip' \ -H "X-Miner-Hotkey: $HOTKEY" \ + -H "X-Lium-Api-Key: $LIUM_API_KEY" \ --data-binary @submission.zip ``` ## JSON (local / scripting) +Same members as the ZIP (or `zip_base64`). Prefer ZIP on live. + ```bash curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ -H 'content-type: application/json' \ + -H "X-Lium-Api-Key: $LIUM_API_KEY" \ -d @submission.json ``` -```json -{ - "miner_hotkey": "<64 lowercase hex>", - "architecture_py": "", - "training_py": "", - "label": "optional human label" -} -``` - -`POST /v1/submissions` is **idempotent** by `submission_id` (digest of hotkey + sources): -re-POSTing identical sources returns `200 {"status":"already-queued"}`. +`POST /v1/submissions` is **idempotent** by `submission_id` (hash of **pin id ‖ +`0x00` ‖ patch bytes**): re-POSTing the identical pin+patch returns +`200 {"status":"already-queued"}`. ## Registration and 1-max gating @@ -51,47 +50,29 @@ re-POSTing identical sources returns `200 {"status":"already-queued"}`. |-----------|----------| | Hotkey not in the subnet metagraph | `403 hotkey_not_in_metagraph` | | Metagraph snapshot not ready yet | `503 metagraph_unavailable` (retry shortly) | -| You already have an accepted architecture submission | `409 submission_gated` | - -**One accepted architecture submission per hotkey.** While yours is -`registered` / `blocked` / `rejected`, a *different* architecture submission gets -`409 submission_gated`. If your hotkey **leaves the metagraph** (uid deregistered or -swapped), the watcher reopens your slot automatically. - -## Training-only entries - -Training-only entries are **separate slots**: one accepted entry per `(hotkey, arch_id)` -— you may train on many published architectures, one script per arch. +| You already have an accepted patch submission | `409 submission_gated` | +| Missing Lium key on live | `400 missing_lium_api_key` | -```bash -# JSON -curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ - -H 'content-type: application/json' \ - -d '{"miner_hotkey":"","arch_id":"","training_py":""}' - -# ZIP (training.py only) + header -curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ - -H 'content-type: application/zip' \ - -H "X-Miner-Hotkey: $HOTKEY" \ - -H "X-Prism-Arch-Id: " \ - --data-binary @training-only.zip -``` +**One accepted patch submission per hotkey.** While yours is `registered` / +`rejected`, or `blocked` **outside** the infra recovery window, a *different* +patch submission gets `409 submission_gated`. Re-POSTing the **identical** +pin+patch is always safe (idempotent). -Do **not** include `architecture.py` in a training-only entry — the source is pulled -from the registry (miner-sent architecture is rejected on these rows). Unknown -`arch_id` → `404 unknown_arch`. +If your hotkey **leaves the metagraph**, the watcher reopens your slot(s) +automatically — resubmit under your new uid. ## Retries and terminal states -- Infra failures (pod provisioning, review/similarity/LLM infra) **auto-retry up to 3 - times**. Retry budget exhausted → `failed`, slot `blocked`. -- Cheat / rejected verdicts are **terminal** — no auto-retry. Manual retry for - infra-class failures: `POST /v1/submissions/{id}/retry`. +- Infra failures (Lium pod, review/similarity/LLM infra) **auto-retry up to 3 + times**. Cheat / rejected verdicts are terminal. +- After an infra failure (`ChallengeInternal`), you may **resubmit within 30 + minutes** (new POST or `POST /v1/submissions/{id}/retry`). After 30 minutes + the slot stays blocked until your hotkey leaves the metagraph. ## Precheck similarity before you submit -Dry-run the same pre-LLM copy gate **without** burning your 1-max slot or a GPU -eval. Same payload as submit (ZIP or JSON): +Dry-run the copy / layout gate **without** burning your 1-max slot or a GPU +eval (send the same AutoModel ZIP you would submit): ```bash curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions/precheck" \ @@ -100,34 +81,30 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions/precheck" \ --data-binary @submission.zip ``` -Example response: - -```json -{ - "similar": false, - "verdict": "clean", - "message": "no earlier architecture copy detected by the pre-LLM gate; full submit still runs similarity + agentic", - "quota": { - "day": "2026-08-08", - "used": 1, - "limit": 3, - "remaining": 2, - "identity": "coldkey" - } -} -``` - | Field | Meaning | |-------|---------| | `similar` | `true` → would hard-reject at intake copy gate | -| `verdict` | `clean` / `copied` / `skipped` (training-only) | +| `verdict` | `clean` / `copied` / `skipped` | | `matched_against` | Corpus id only (never competitor source) | | `score` | Similarity in `[0,1]` when compared | -| `quota` | Daily budget (`limit` = 3) | +| `quota` | `{ day, used, limit: 3, remaining, identity }` | + +**Quota: 3 attempts per coldkey per UTC day** (falls back to hotkey when the +metagraph Owner coldkey is unknown). Rotating hotkeys under the same coldkey +does **not** reset the budget. A 4th call returns `429` / +`precheck_quota_exceeded` with `remaining=0`. Precheck never creates a scored +submission and never rents a Lium pod. + +## Inspect your applied diff + +After intake: + +```bash +curl -sS "$GATEWAY/challenge/prism/v1/submissions/$SUB/diff" +``` -**Quota: 3 attempts per coldkey per UTC day** (hotkey fallback when Owner is -unknown). Rotating hotkeys under the same coldkey does **not** reset the budget. -A 4th call returns `429` / `precheck_quota_exceeded` with `remaining=0`. +Returns the full unified diff plus diffstat / classification +(`arch` / `trainer` / `data` / `other`). ## Gateways diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 1051754..7f9dd05 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,25 +1,34 @@ # Troubleshooting -| Symptom | What to check | -|---------|----------------| -| `400` on submit | Contract shape: `def build_model(` in `architecture.py`, `def train(` in `training.py`, ≤ 128 KiB per script | -| `403 hotkey_not_in_metagraph` | Hotkey not registered on the subnet; check the hex (64 lowercase, no `0x`) | -| `404 unknown_arch` | Training-only `arch_id` not in the registry — `GET /v1/architectures` | -| `409 submission_gated` | You already have an accepted submission (1-max per hotkey, per `(hotkey, arch_id)` for training-only) | -| `503 metagraph_unavailable` | Snapshot lag after a fresh registration — retry in a couple of minutes | -| `missing_telemetry_hooks` | `training.py` must import `prism_telemetry`, call `report(...)`, and may call `finish_evaluation()` | -| Status `rejected` | Pre-LLM copy gate: your `architecture.py` matches an *earlier* one. Write your own; start from the baseline | -| `similar: true` on precheck | Would hit intake copy gate — revise before `POST /v1/submissions` | -| `429 precheck_quota_exceeded` | 3 prechecks/coldkey/UTC day used; rotating hotkeys does not reset | -| `auto_retry` events | Infra failure retrying (up to 3 retries) — wait for the final state | -| `failed` after retries | Slot `blocked`; leaving the metagraph reopens it via the watcher | -| No telemetry on the site | Your loop never called `report(...)` — hooks are mandatory | -| Wrong host | Use the **gateway** `/challenge/prism/...` prefix (prod `https://chain.joinbase.ai`) | +| Symptom | Likely cause | What to check | +|---------|--------------|---------------| +| Rejected submit | Recipe contract | `GET /v1/recipe` (`automodel_pin_id` + caps); follow recipe **2.0** | +| `400 unsupported_layout` | Legacy 1.x ZIP or missing AutoModel members | Ship `automodel.base` + `automodel.patch` (+ optional `prism.toml`). Two-script / source-tree / `arch_id` layouts are rejected on live 2.0 | +| `400 recipe_version` | Payload implies recipe 1.x while live advertises ≥ 2.0 | Re-pack as AutoModel patch ZIP; do not send `architecture.py`/`training.py` | +| Patch apply failure / conflict | Diff not against the live pin, or stale rebase | Checkout exact `automodel_git_commit` from `/v1/recipe`; regenerate `git diff `; ensure `automodel.base` == `automodel_pin_id` | +| Wrong / unknown pin id | `automodel.base` ≠ recipe `automodel_pin_id` | Copy `automodel_pin_id` (live: `automodel@v0.5.0`) byte-identical from `/v1/recipe` | +| Binary / path-escape / oversized patch | Fail-closed apply rules | Text-only unified diff; no path escape outside allowlisted roots; keep diff within intake budgets | +| Tokenizer / hub errors on pod | No network; Hub download from miner code | Stay offline; use pin/harness tokenizer paths — do not `from_pretrained("")` | +| `CAP_EXCEEDED` / Score 0 | Model > 350M params | Terminal — resize model config in your patch; not auto-retried | +| `missing_telemetry_hooks` | Patch removed / bypassed harness telemetry | Keep `prism_telemetry.report` (+ optional `finish_evaluation`) under the AutoModel train entry | +| Score 0 after review | `Copied` / high-confidence `Suspicious` (≥0.9, non-trope) | Similarity on **your delta**; rewrite unique hunks; tropes alone are not plagiarism | +| `similar: true` on precheck | Would hit intake copy gate | Change the patch vs prior champions; starting from the operator pin is fine | +| `429 precheck_quota_exceeded` | 3 prechecks/coldkey/UTC day used | Wait until next UTC day; rotating hotkeys does not reset | +| `400 missing_lium_api_key` | Live path needs miner-funded Lium | Pass `X-Lium-Api-Key` (your Lium account); see [Submit](submit.md) | +| `403 hotkey_not_in_metagraph` | Hotkey not registered | Check the hex (64 lowercase, no `0x`) | +| `409 submission_gated` | 1-max slot already used | One accepted patch per hotkey; identical pin+patch is idempotent | +| `503 metagraph_unavailable` | Snapshot lag after a fresh registration | Retry in a couple of minutes | +| Stuck `Provisioning` | Lium market / underfunded key | Check your Lium balance; watch `GET /v1/jobs` / events | +| Idempotent replay | Same `submission_id` (pin id + patch bytes) | Expected — returns prior row | +| Wrong host | Not using gateway prefix | Use **gateway** `/challenge/prism/...` (prod `https://chain.joinbase.ai`) | When something fails, poll `events` — status alone is rarely enough: ```bash curl -sS "$GATEWAY/challenge/prism/v1/submissions/$SUB/events" +curl -sS "$GATEWAY/challenge/prism/v1/submissions/$SUB/diff" ``` -Frozen contracts live in the BASE monorepo: `docs/PRISM.md` + `docs/PRISM_RECIPE.md`. +Frozen contracts live in the BASE monorepo: +[`docs/PRISM.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM.md) + +[`docs/PRISM_RECIPE.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM_RECIPE.md). diff --git a/examples/baseline/README.md b/examples/baseline/README.md index b9b85e5..d2ce870 100644 --- a/examples/baseline/README.md +++ b/examples/baseline/README.md @@ -1,54 +1,28 @@ -# PRISM baseline recipe (example) +# Historical recipe 1.x baseline (not accepted on live 2.0) -The official baseline submission — a tiny GPT-style causal transformer (~12M params) -plus an AdamW training loop that demonstrates the **required telemetry hooks**. +> **Live recipe is 2.0.0.** Submitting `architecture.py` + `training.py` returns +> `400 unsupported_layout` / `recipe_version` on production. Use the AutoModel +> patch workflow instead — see [Getting started](../../docs/getting-started.md) +> and the [full guide](../../docs/prism.md). -Starting from this baseline is always allowed (the anti-copy gate exempts it). To be -competitive, ship your **own** architecture — see -[scoring & competition](../../docs/scoring.md). +This directory keeps the old two-script baseline for reference only (tiny +GPT-style causal transformer + AdamW loop with telemetry hooks). It is **not** +a valid live submission under recipe 2.0. -## Files +## Live path (recipe 2.0) -| File | Role | -|------|------| -| `architecture.py` | `build_model(ctx)` → TinyGPT (tied embeddings, causal mask, block=512) | -| `training.py` | `train(model, ctx)` → AdamW loop with `prism_telemetry.report(...)` + `finish_evaluation()` | - -## Hook pattern (required since recipe 1.1.0) - -`training.py` imports `prism_telemetry` with a local fallback stub so you can also run -it outside the operator harness: - -```python -try: - import prism_telemetry -except ImportError: - class _TelemetryFallback: - @staticmethod - def report(**_kwargs): ... - @staticmethod - def finish_evaluation(): ... - prism_telemetry = _TelemetryFallback() -``` +1. `GET /v1/recipe` → copy `automodel_pin_id` (`automodel@v0.5.0`) and + `automodel_git_commit`. +2. Checkout that AutoModel commit, edit, `git diff > automodel.patch`. +3. Pack `automodel.base` + `automodel.patch` (+ optional `prism.toml`). +4. `POST /v1/submissions` with `X-Miner-Hotkey` + **`X-Lium-Api-Key`**. -Inside the operator harness the real module captures your series into -`METRICS_JSON.telemetry.loss_series` (persisted master-side; served at -`/v1/site/arenas/prism/submissions/{id}/telemetry`). +## Files (legacy 1.x) -- `prism_telemetry.report(loss=..., step=..., grad_norm=..., layer_stats=...)` — call - every N steps. -- `prism_telemetry.finish_evaluation()` — optional early stop; the harness scores the - in-memory model as-is. It raises a `BaseException` through `train()`, so it cannot be - swallowed by your own `except Exception` blocks. - -## Submit - -```bash -zip -j submission.zip architecture.py training.py -curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ - -H 'content-type: application/zip' \ - -H "X-Miner-Hotkey: $HOTKEY" \ - --data-binary @submission.zip -``` +| File | Role | +|------|------| +| `architecture.py` | `build_model(ctx)` → TinyGPT (historical) | +| `training.py` | `train(model, ctx)` → AdamW + `prism_telemetry` hooks (historical) | -The same sources are always available live at `GET /v1/recipe/baseline`. +Telemetry hooks remain required under the AutoModel train entry on live 2.0 — +patches that remove them fail with `missing_telemetry_hooks`.