-
Notifications
You must be signed in to change notification settings - Fork 1
docs: sync AutoModel recipe 2.0 miner guide #9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,7 @@ | |
|
|
||
| # PRISM | ||
|
|
||
| **Miner guide for the BASE prism challenge — HTTP recipe submit.** | ||
| **Miner guide for the BASE prism challenge — HTTP AutoModel patch submit.** | ||
|
|
||
| [](https://github.com/BaseIntelligence/base) | ||
| [](https://bittensor.com/) | ||
|
|
@@ -15,59 +15,63 @@ | |
| [Submit](docs/submit.md) · | ||
| [Scoring & competition](docs/scoring.md) · | ||
| [API](docs/api.md) · | ||
| [Examples](examples/baseline/) | ||
| [Full guide](docs/prism.md) | ||
|
|
||
| </div> | ||
|
|
||
| ## What it is | ||
|
|
||
| PRISM is a research challenge: you try **new architectures** and the challenge re-executes | ||
| them fairly. You submit **two Python scripts** — `architecture.py` (`build_model(ctx)`) and | ||
| `training.py` (`train(model, ctx)`) — and the operator runs them on a GPU pod against a | ||
| pinned FineWeb-Edu shard. Score is pure **bits-per-byte** (bpb, lower is better) measured | ||
| by the operator harness. There is **no** miner Docker image, no CVM, no on-chain write from | ||
| miners — HTTP submit only. | ||
| PRISM is a research challenge on a pinned | ||
| [NeMo AutoModel](https://github.com/NVIDIA-NeMo/Automodel) base: you fork the | ||
| operator pin, edit under that tree, and submit a **unified git diff**. The | ||
| operator applies your patch fail-closed, then re-executes training on a | ||
| miner-funded Lium GPU pod against a pinned FineWeb-Edu shard. Score is pure | ||
| **bits-per-byte** (bpb, lower is better). There is **no** miner Docker image, | ||
| no CVM, no on-chain write from miners — HTTP submit only. | ||
|
|
||
| | | | | ||
| |---|---| | ||
| | Challenge id | `prism` | | ||
| | Production gateway | `https://chain.joinbase.ai` | | ||
| | Staging gateway | `http://staging.api.joinbase.ai` | | ||
| | Submit path | `/challenge/prism/v1/submissions` | | ||
| | Recipe | v1.2.0 — telemetry hooks required | | ||
| | Recipe | **2.0.0** — AutoModel pin + patch (`automodel@v0.5.0`) | | ||
| | Live GPU | Miner-funded Lium — pass `X-Lium-Api-Key` | | ||
|
|
||
| This repository holds **miner documentation and examples only**. Control-plane source | ||
| lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). | ||
| This repository holds **miner documentation and examples only**. Control-plane | ||
| source lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). | ||
|
|
||
| ## Start here | ||
|
|
||
| 1. Read [Getting started](docs/getting-started.md). | ||
| 2. Copy [`examples/baseline/`](examples/baseline/) — it shows the required telemetry | ||
| hooks (`prism_telemetry.report` + `finish_evaluation`). | ||
| 3. Zip `architecture.py` + `training.py` and submit — see [Submit](docs/submit.md). | ||
| 4. Poll events until `terminated`, then check your bpb — see [API](docs/api.md). | ||
| 1. Read [Getting started](docs/getting-started.md) (or the [full guide](docs/prism.md)). | ||
| 2. `GET /v1/recipe` — copy `automodel_pin_id`, `automodel_git_commit`, and caps. | ||
| 3. Checkout that AutoModel commit → edit → `git diff <commit> > automodel.patch`. | ||
| 4. Pack `automodel.base` + `automodel.patch` (+ optional `prism.toml`) and submit | ||
| with your hotkey + **`X-Lium-Api-Key`** — see [Submit](docs/submit.md). | ||
| 5. Poll events until `terminated`, then check your bpb — see [API](docs/api.md). | ||
|
|
||
| ```bash | ||
| export GATEWAY=https://chain.joinbase.ai | ||
| export HOTKEY=<64 lowercase hex> # public hotkey only — never a secret key | ||
| export LIUM_API_KEY=<your Lium API key> | ||
|
|
||
| cd examples/baseline | ||
| zip -j submission.zip architecture.py training.py | ||
| # After forking the pin and producing automodel.base + automodel.patch: | ||
| zip -j submission.zip automodel.base automodel.patch # + prism.toml if used | ||
|
|
||
|
Comment on lines
+58
to
60
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Ensure the documented package command includes optional
📍 Affects 2 files
🤖 Prompt for AI Agents |
||
| curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ | ||
| -H 'content-type: application/zip' \ | ||
| -H "X-Miner-Hotkey: $HOTKEY" \ | ||
| -H "X-Lium-Api-Key: $LIUM_API_KEY" \ | ||
| --data-binary @submission.zip | ||
| ``` | ||
|
|
||
| ## The three things miners get wrong | ||
|
|
||
| 1. **Missing telemetry hooks** — `training.py` must import `prism_telemetry` and call | ||
| `report(...)` during training (and may call `finish_evaluation()` to stop early). | ||
| Missing hooks = hard contract violation, zero score, terminal. | ||
| 2. **Copying someone's `architecture.py`** — the pre-GPU copy gate rejects byte/AST | ||
| copies of *earlier* architectures with zero score, no appeal. Starting from the | ||
| published baseline is fine. | ||
| 3. **Submitting again while gated** — one accepted architecture submission per hotkey; | ||
| a second one returns `409 submission_gated`. Training-only entries on published | ||
| architectures are separate slots (one per `(hotkey, arch_id)`). | ||
| 1. **Legacy 1.x ZIPs** — `architecture.py` + `training.py` (or training-only | ||
| `arch_id`) return `400 unsupported_layout` / `recipe_version` on live 2.0. | ||
| Ship `automodel.base` + `automodel.patch` only. | ||
| 2. **Wrong pin / stale diff** — `automodel.base` must equal live | ||
| `automodel_pin_id` (`automodel@v0.5.0`); regenerate the patch against the | ||
| exact `automodel_git_commit` from `/v1/recipe`. | ||
| 3. **Missing `X-Lium-Api-Key`** — live eval runs on **your** Lium account. | ||
| Missing key → `400 missing_lium_api_key`. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,18 @@ | ||
| # PRISM miner docs | ||
|
|
||
| Live recipe is **2.0.0**: submit an AutoModel pin id + unified git diff | ||
| (`automodel.base` + `automodel.patch`), not a free-form two-script ZIP. | ||
|
|
||
| | Page | What it covers | | ||
| |------|----------------| | ||
| | [Getting started](getting-started.md) | Contract, hooks, dataset, budgets | | ||
| | [Submit](submit.md) | ZIP/JSON submit, gating, retries, training-only entries | | ||
| | [Scoring & competition](scoring.md) | bpb lattice, anti-copy, architecture competition, top-model | | ||
| | [API](api.md) | Routes, statuses, telemetry | | ||
| | [Troubleshooting](troubleshooting.md) | Common failures | | ||
| | [`examples/baseline/`](../examples/baseline/) | Reference recipe with hooks | | ||
| | [Getting started](getting-started.md) | Fork pin → edit → `git diff` → pack ZIP | | ||
| | [Submit](submit.md) | ZIP/JSON, BYOK Lium key, gating, precheck, retries | | ||
| | [Scoring & competition](scoring.md) | bpb lattice, patch anti-copy, causal ban, top-model | | ||
| | [API](api.md) | Routes, statuses, diff + telemetry | | ||
| | [Troubleshooting](troubleshooting.md) | `unsupported_layout`, pin/patch failures, Lium | | ||
| | [Full guide](prism.md) | Complete miner guide (mirrors BASE `docs/external-miner/prism.md`) | | ||
|
|
||
| Normative sources (BASE monorepo): `docs/PRISM.md`, `docs/PRISM_RECIPE.md`, | ||
| `docs/external-miner/prism.md`. | ||
| Normative sources (BASE monorepo): | ||
| [`docs/PRISM.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM.md), | ||
| [`docs/PRISM_RECIPE.md`](https://github.com/BaseIntelligence/base/blob/main/docs/PRISM_RECIPE.md), | ||
| [`docs/external-miner/prism.md`](https://github.com/BaseIntelligence/base/blob/main/docs/external-miner/prism.md). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,54 +1,57 @@ | ||
| # Getting started | ||
|
|
||
| ## The contract (recipe v1.2.0) | ||
| ## The contract (recipe v2.0.0) | ||
|
|
||
| You ship **two scripts only**. The operator harness (`prism_harness.py`) imports them, | ||
| downloads the pinned dataset, verifies its SHA-256, times the run, and reports | ||
| `METRICS_JSON` (bpb, tokens, steps, wall clock, gpu, params). | ||
| You do **not** ship a free-form `architecture.py` / `training.py` project. | ||
| Live recipe **2.0.0** accepts only a pin id plus your unified diff against that | ||
| pin: | ||
|
|
||
| ```python | ||
| # architecture.py | ||
| def build_model(ctx): | ||
| """Return a model given the recipe context (devices, dims, seed).""" | ||
|
|
||
| # training.py | ||
| def train(model, ctx): | ||
| """Train the model; must respect ctx.budget(): | ||
| budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train).""" | ||
| ```text | ||
| automodel.base # required — pin id from GET /v1/recipe (live: automodel@v0.5.0) | ||
| automodel.patch # required — unified diff vs that pin (git diff pin...HEAD) | ||
| prism.toml # optional — entry / model-config knobs | ||
| ``` | ||
|
|
||
| No third source file, no offline weights, no network at pod runtime beyond the pinned | ||
| dataset pull. | ||
| **Workflow: fork pin → edit → `git diff` → submit** | ||
|
|
||
| ## Telemetry hooks (required since recipe 1.1.0) | ||
| 1. Read the live pin from `GET /v1/recipe` (`automodel_pin_id`, | ||
| `automodel_repo_url`, `automodel_git_commit`, `automodel_content_sha256`). | ||
| 2. Check out that exact AutoModel commit (or extract the staged archive and | ||
| verify `automodel_content_sha256` matches `/v1/recipe`). | ||
| 3. Edit under the AutoModel layout — new model modules / configs are allowed; | ||
| trainer / data-path edits get high scrutiny. | ||
| 4. Produce a unified diff against the pin commit, e.g. | ||
| `git diff <automodel_git_commit> > automodel.patch`. | ||
| 5. Write `automodel.base` as a single line equal to `automodel_pin_id`, pack | ||
| the ZIP, and `POST /v1/submissions` with your hotkey + **`X-Lium-Api-Key`**. | ||
|
|
||
| The harness registers a `prism_telemetry` module before your code loads (also at | ||
| `ctx["telemetry"]`). Your `training.py` **MUST**: | ||
| Models must stay **≤ 350M parameters**. The pod has **no network** | ||
| (`unshare --net`) beyond the operator-owned dataset pull — do not call Hub | ||
| downloads from miner code. | ||
|
|
||
| ```python | ||
| import prism_telemetry | ||
| **Legacy recipe 1.x is rejected on live.** Two-script ZIPs | ||
| (`architecture.py` + `training.py`), 1.3 source-tree ZIPs, and training-only | ||
| `arch_id` submissions return `400 unsupported_layout` or `400 recipe_version`. | ||
| Do not ship Megatron-Bridge or other non-AutoModel frameworks. | ||
|
|
||
| prism_telemetry.report(loss=..., step=..., grad_norm=..., layer_stats=...) # every N steps | ||
| prism_telemetry.finish_evaluation() # optional early stop: score the model as-is | ||
| ``` | ||
| ## Pay for your own GPU (required on live) | ||
|
|
||
| - `report(...)` feeds the loss/gradient/layer series persisted master-side and shown on | ||
| the site (`/v1/site/arenas/prism/submissions/{id}/telemetry`). | ||
| - `finish_evaluation()` raises a `BaseException` through `train()` so your own | ||
| `except Exception` blocks cannot swallow it; without it the eval ends when `train()` | ||
| returns or the wall-clock cap fires. | ||
| - **Missing hooks are a hard contract violation**: the review fails the submission | ||
| (`missing_telemetry_hooks`, zero score, terminal — no retry). | ||
| Create a [Lium](https://lium.io) account, fund it, and pass your API key on | ||
| every live submit: | ||
|
|
||
| ```http | ||
| X-Lium-Api-Key: <your Lium API key> | ||
| ``` | ||
|
|
||
| The [baseline example](../examples/baseline/) shows the exact pattern, including an | ||
| offline fallback stub for local testing. | ||
| The key is held only in master memory for that submission (never stored in the | ||
| DB, never logged). Missing key on live → `400 missing_lium_api_key`. | ||
|
|
||
| ## Training-only submissions (architecture competition, recipe ≥ 1.2.0) | ||
| ## Telemetry hooks (still required) | ||
|
|
||
| Instead of shipping both scripts you can submit `training.py` + `arch_id` referencing a | ||
| **published** architecture. The master pulls `architecture.py` from the registry; the | ||
| same harness contract applies unchanged. Published archs: `GET /v1/architectures`. | ||
| See [Submit](submit.md#training-only-entries). | ||
| The harness wrap still requires `prism_telemetry` reporting / | ||
| `finish_evaluation` under the AutoModel train entry. Patches that remove or | ||
| bypass those hooks fail review (`missing_telemetry_hooks`, zero score, | ||
| terminal). | ||
|
Comment on lines
+49
to
+54
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- affected documentation ---'
for f in docs/getting-started.md docs/prism.md docs/troubleshooting.md examples/baseline/README.md; do
echo "### $f"
sed -n '1,180p' "$f" | nl -ba | sed -n '1,180p'
done
printf '%s\n' '--- telemetry references ---'
rg -n -i -C 3 'prism_telemetry|finish_evaluation|missing_telemetry_hooks|AutoModel|telemetry' . \
-g '!node_modules' -g '!dist' -g '!build'
printf '%s\n' '--- repository files ---'
git ls-files | sed -n '1,240p'Repository: BaseIntelligence/prism Length of output: 30008 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- exact affected sections ---'
for spec in \
'docs/getting-started.md:45:56' \
'docs/prism.md:43:56' \
'docs/troubleshooting.md:1:15' \
'examples/baseline/README.md:1:30' \
'examples/baseline/training.py:1:100' \
'top-model/training.py:1:96' \
'docs/scoring.md:59:70'; do
file=${spec%%:*}; rest=${spec#*:}; start=${rest%%:*}; end=${rest##*:}
echo "### $file:$start-$end"
sed -n "${start},${end}p" "$file" | awk '{printf "%6d %s\n", NR + '"$start"' - 1, $0}'
done
printf '%s\n' '--- all exact telemetry contract statements ---'
rg -n -C 2 'required|optional|either way|finish_evaluation|missing_telemetry_hooks' \
docs examples top-model README.md
printf '%s\n' '--- executable repository/config references to the error name ---'
rg -n 'missing_telemetry_hooks|telemetry_hooks|finish_evaluation|prism_telemetry' \
.github docs examples top-model README.mdRepository: BaseIntelligence/prism Length of output: 24646 🏁 Script executed: #!/bin/bash
set -e
python3 - <<'PY'
import ast
from pathlib import Path
docs = [
"docs/getting-started.md",
"docs/prism.md",
"docs/troubleshooting.md",
"examples/baseline/README.md",
]
terms = ("prism_telemetry", "finish_evaluation", "missing_telemetry_hooks")
for name in docs:
text = Path(name).read_text()
hits = []
for i, line in enumerate(text.splitlines(), 1):
if any(term in line for term in terms):
hits.append((i, line.strip()))
print(f"### {name}")
for line_no, line in hits:
print(f"{line_no}: {line}")
for name in ("examples/baseline/training.py", "top-model/training.py"):
tree = ast.parse(Path(name).read_text(), filename=name)
train = next(
node for node in tree.body
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == "train"
)
calls = []
for node in ast.walk(train):
if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
if isinstance(node.func.value, ast.Name) and node.func.value.id == "prism_telemetry":
calls.append(node.func.attr)
print(f"### {name} train() telemetry calls")
print(", ".join(calls) or "(none)")
PYRepository: BaseIntelligence/prism Length of output: 1366 Use one authoritative telemetry-hook contract. Keep 📍 Affects 4 files
🤖 Prompt for AI Agents |
||
|
|
||
| ## Pinned dataset | ||
|
|
||
|
|
@@ -60,43 +63,32 @@ See [Submit](submit.md#training-only-entries). | |
| | SHA-256 | `e5a2eae25f057f0856a10bfae314c6ca8ea8bb08456d2131e9e89b2b8305e2f6` | | ||
|
|
||
| The harness re-verifies the hash on the file it actually fetched; a mismatch ends the | ||
| eval as `ChallengeInternal` — never a miner score. | ||
| eval as `ChallengeInternal` — never a miner score. Always confirm live values via | ||
| `GET /v1/recipe`. | ||
|
|
||
| ## Budget & caps | ||
|
|
||
| | Cap | Value | | ||
| |-----|-------| | ||
| | Train wall clock | 6.0 h per submission | | ||
| | Pod lifetime | 7.0 h (train + bootstrap margin) | | ||
| | Hard step cap | 20 000 | | ||
| | Source size | 128 KiB per script | | ||
| | Model parameters | ≤ **350 000 000** after `build_model` | | ||
| | `train_rows` (from `GET /v1/recipe`) | **2048** — baseline / default cut in `ctx` | | ||
| | `val_rows` | **256** — frozen val scored by the harness | | ||
|
|
||
| `train_rows` is what the **sealed baseline** trains on (~2M GPT-2 tokens for | ||
| that slice). It is **not** a hard “you only get 2048 rows” ceiling for | ||
| competitive recipes: the harness gives you the full pinned parquet at | ||
| `ctx["dataset_path"]`, and you may stream it until the 6h / 20k-step guard | ||
| fires. Token count then depends on your loop and the GPU — a long Lium run can | ||
| reach ~O(10⁹) tokens. Marketing charts that once said “2.6B tokens · single | ||
| pass” were showing a leader’s **observed** telemetry, not a fixed recipe | ||
| quota. Always trust live `GET /v1/recipe` (`pin_hex`, `train_rows`, caps). | ||
|
|
||
| The sealed baseline is deliberately mediocre (short cut, few steps). Matching | ||
| a board BPB near ~4–5 requires a competitive trainer, not an unmodified | ||
| baseline on a 4090 for a few minutes. | ||
| | Train wall clock | 6.0 h per submission (`train_hours_cap`) | | ||
| | Hard step cap | 20 000 (`max_train_steps`) | | ||
| | Model parameters | ≤ **350 000 000** (`max_params`) | | ||
|
|
||
| Trust live `GET /v1/recipe` (`version`, `automodel_*`, `pin_hex`, caps) over any | ||
| marketing chart. | ||
|
|
||
| ## Recipe pin | ||
|
|
||
| `GET /v1/recipe` returns the versioned descriptor (dataset URL/hash, caps, | ||
| `train_rows` / `val_rows`, recipe version, `pin_hex`). Production today is | ||
| recipe **1.2.0** — open docs PRs that advertise 1.3+/1.4.0/v3 scoring describe | ||
| **unreleased** control-plane work (`prism-better`), not what | ||
| `https://chain.joinbase.ai` executes. `GET /v1/recipe/baseline` returns the | ||
| official baseline scripts — the best starting point for your own architecture. | ||
| ```bash | ||
| curl -sS "$GATEWAY/challenge/prism/v1/recipe" | ||
| ``` | ||
|
|
||
| Live recipe **2.0.0** advertises `version: "2.0.0"` and AutoModel pin fields | ||
| (`automodel_pin_id` = `automodel@v0.5.0`, `automodel_repo_url`, | ||
| `automodel_git_ref`, `automodel_git_commit`, `automodel_content_sha256`). | ||
|
|
||
| ## Next | ||
|
|
||
| → [Submit](submit.md) | ||
| → [Full guide](prism.md) | ||
| → [Scoring & competition](scoring.md) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not publish a cleartext staging gateway for credential-bearing requests.
README.md#L34-L39: Replace the HTTP staging endpoint or document a staging flow that never sends a realX-Lium-Api-Key.docs/submit.md#L109-L116: Use an HTTPS staging base URL or separate staging authentication from the live BYOK workflow.docs/api.md#L3-L6: Update the staging gateway documentation so clients do not reuse credential-bearing commands over HTTP.📍 Affects 3 files
README.md#L34-L39(this comment)docs/submit.md#L109-L116docs/api.md#L3-L6🤖 Prompt for AI Agents