Quickstart: full knowledge loop over 3 hard ARC-AGI-1 tasks

README.md

@@ -41,10 +41,12 @@ bash scripts/quickstart.sh
 
 The script self-bootstraps everything it needs: it synthesizes a provider
 profile from your key, builds the `ksi-agent:bench` image on first run,
-installs the host Node dependencies, then runs one generation over the
-bundled [`examples/custom_tasks/`](./examples/custom_tasks/) demo. If
-anything is missing, `uv run ksi-doctor` prints a ✓/✗ readiness checklist
-with the exact command to fix it.
+installs the host Node dependencies, then runs three generations over three
+hard [ARC-AGI-1](./examples/quickstart/arc1_hard/) tasks (bundled, no download)
+— with the forums on, so the full execute → forum → distill → seed loop fires.
+ARC tasks are hard for every current model, so they don't all solve on the first
+generation and the loop keeps going. If anything is missing, `uv run ksi-doctor`
+prints a ✓/✗ readiness checklist with the exact command to fix it.
 
 ## Documentation
 

docs/experiments.md

@@ -1,9 +1,9 @@
 # Running larger runs
 
 The [quickstart](getting-started.md) and [your own tasks](your_own_tasks.md)
-walkthroughs run one generation over a handful of tiny tasks. This page
-covers the flags that matter once you scale up — more tasks, more
-generations, or a maintained reference benchmark instead of your own tasks.
+walkthroughs run over a handful of tiny tasks. This page covers the flags
+that matter once you scale up — more tasks, more generations, or a
+maintained reference benchmark instead of your own tasks.
 
 There is **one canonical launch surface**: the `ksi.cli` argument parser.
 Everything else (bash presets, `ksi.run(...)`) is a layer over the same

docs/faq.md

@@ -83,8 +83,9 @@ Copy the template for your provider, fill in your key, and pass the path with
 Every run makes real LLM API calls billed to the key in your provider profile;
 there is no built-in spending cap. Cost scales with the number of tasks,
 generations, and the model you choose. To get a feel before committing,
-start with the bundled synthetic demo: `bash scripts/quickstart.sh` runs
-three tasks, one generation, with Haiku — the fastest and cheapest option.
+start with the bundled demo: `bash scripts/quickstart.sh` runs three hard
+ARC-AGI-1 tasks across three generations with Haiku — the cheapest way to
+watch the full loop.
 Use `DRY_RUN=true` on any experiment wrapper script to print the full CLI
 command and DB paths without launching anything.
 

docs/getting-started.md

@@ -1,12 +1,15 @@
 # Getting started
 
-Go from a fresh clone to a solved demo task in one command, then learn what just happened.
+Go from a fresh clone to the full knowledge loop running in one command, then learn what just happened.
 
 ## What you'll do
 
-Run one fast generation of agents against three bundled, self-contained tasks and see them score —
-no dataset download, no manual setup. The whole demo finishes in a few minutes and leaves you with
-a working environment ready to run your own tasks or a reference benchmark.
+Run three generations of agents against five bundled **ARC-AGI-1** tasks — hard enough for any
+current model that they don't all solve on the first try — and watch the full knowledge loop —
+execute, discuss, distill, seed — fire end to end. No dataset download, no manual setup (the ARC-1
+tasks are vendored under `benchmarks/arc1/`). ARC attempts are slow, so a full three-generation
+run takes on the order of 15–20 minutes; it leaves you with a working environment ready to run your
+own tasks or a reference benchmark.
 
 ## Prerequisites
 
@@ -24,10 +27,14 @@ bash scripts/quickstart.sh
 
 The script self-bootstraps everything it needs: it synthesizes a provider profile from your key,
 builds the `ksi-agent:bench` image on first run (this takes a few minutes), installs the host
-Node dependencies, then runs one generation over the three bundled tasks under
-[`examples/custom_tasks/`](https://github.com/recursive-knowledge/KSI/tree/main/examples/custom_tasks)
-(`fizzbuzz`, `reverse-words`, `anagram-groups`) — each graded by running `python3 tests.py`
-against the agent's attempt.
+Node dependencies, then runs three generations over three hard **ARC-AGI-1** tasks — bundled under
+[`examples/quickstart/arc1_hard/`](https://github.com/recursive-knowledge/KSI/tree/main/examples/quickstart/arc1_hard)
+(copied from the ARC-1 corpus vendored under `benchmarks/arc1/`, no download) — with the per-task
+and cross-task forums on so every phase of the loop fires. Each agent studies an ARC task's
+input→output training examples and writes its predicted grids, scored by the `arc_session`
+evaluator (exact-match, up to two attempts per test). These three tasks are chosen for being hard
+for current models (see the [directory README](https://github.com/recursive-knowledge/KSI/blob/main/examples/quickstart/arc1_hard/README.md)
+for their observed pass rates), so they don't all solve on the first generation.
 
 For the complete benchmark environment (including benchmark preparation and
 smoke tests), run `bash scripts/setup_all.sh`. Use `--no-test` when you need
@@ -60,10 +67,16 @@ The run logs each attempt and its score as it progresses. When it finishes, resu
 | Score summary (optional — only when `--output-json` is set) | `results/<experiment>.json` |
 | Execution traces | `analysis/traces/<experiment>/` |
 
-For the quickstart, `<experiment>` defaults to `quickstart_demo`. The run prints
-each task's score as it goes and ends with a `completed … solved=3/3 (100.0%)`
-line — that's the signal your environment is set up correctly. Elapsed times and
-token counts vary by model and run; the task names and `solved=3/3` don't.
+For the quickstart, `<experiment>` defaults to `quickstart_demo`. The run logs
+each task's score (ARC scoring is exact-match: `1.0` solved, `0.0` not) as it
+goes, and ends with a single `completed traces=… tasks=… solved=N/M` summary
+line. These ARC tasks are hard for current models, so expect at least one to
+remain unsolved after generation 1 (a strong model may solve the rest); the
+unsolved tasks carry forward and get re-attempted each generation, now seeded
+with what the population distilled. **The signal that your environment is set up
+correctly is that attempts run and get scored at all** — not that everything
+solves. Whether an unsolved task flips to solved by generation 3 depends on the
+model. Elapsed times, token counts, and solve counts vary by model and run.
 
 ??? note "A closer look — sample output, optional artifacts, and the knowledge DB"
 
@@ -72,30 +85,50 @@ token counts vary by model and run; the task names and `solved=3/3` don't.
     run preset, which sets it for you) for a score summary on disk. Traces default
     to `analysis/traces/<experiment>/` — set `KSI_TRACE_DIR` to change the root.
 
-    A real excerpt from a run against `claude-haiku-4-5-20251001` (the default
-    `configs/ksi/.env.haiku` profile), timestamps trimmed:
+    An excerpt from a real run (`gpt-5.4-mini`, timestamps trimmed). ARC scores
+    are binary (exact-match); a strong model solves some tasks on generation 1
+    while the hardest carry forward:
 
     ```text
-    INFO ksi.orchestrator.execution_phase: [gen 1] task=reverse-words agent=agent-1 done elapsed=27.4s score=1.0000
-    INFO ksi.orchestrator.execution_phase: [gen 1] task=fizzbuzz agent=agent-0 done elapsed=28.1s score=1.0000
-    INFO ksi.orchestrator.execution_phase: [gen 1] task=anagram-groups agent=agent-2 done elapsed=33.0s score=1.0000
-    INFO ksi.orchestrator.engine: completed traces=3 tasks=3 solved=3/3 (100.0%)
-    INFO ksi.orchestrator.persistence: [tokens] total=418,329 cached_input=346,149 uncached_input=9,129 output=5,090 cache_create=57,961
+    INFO ksi.orchestrator.execution_phase: [gen 1] task=776ffc46 agent=agent-0 done elapsed=281.6s score=1.0000
+    INFO ksi.orchestrator.execution_phase: [gen 1] task=97239e3d agent=agent-1 done elapsed=281.6s score=1.0000
+    INFO ksi.orchestrator.execution_phase: [gen 1] task=d22278a0 agent=agent-2 done elapsed=287.4s score=0.0000
+    INFO ksi.orchestrator.distillation_phase: [ENGINE] distill gen=1: 1 per-task bundle(s), cross_task=0
+    INFO ksi.orchestrator.persistence: [gen 2] start agents=1
     ```
 
-    **Knowledge DB check** — every solved attempt writes an `entry_type='attempt'`
-    row plus an `insight` row. The quickstart turns both forums off for speed
-    (`--per-task-forum-rounds 0 --cross-task-forum-rounds 0`), so there are no
-    discussion posts, and with nothing unsolved in this single-generation run
-    distillation has nothing to write either:
+    Here two tasks solved and dropped out (`--drop-solved`), while `d22278a0`
+    stayed unsolved and carried through generations 2 and 3 — each time
+    re-attempted with freshly distilled guidance seeded in. The run ended with
+    `completed traces=5 tasks=3 solved=2/3 (66.7%)`: five attempts across three
+    generations over three unique tasks.
+
+    **Knowledge DB check** — because the demo runs the full loop, the knowledge
+    DB carries rows from every phase, not just execution. Group by `entry_type`
+    and `source_phase` to see them (real counts from the run above):
 
     ```console
     $ sqlite3 runtime_state/knowledge/quickstart_demo/quickstart_demo_knowledge.sqlite \
         "select entry_type, source_phase, count(*) from knowledge group by entry_type, source_phase order by entry_type, source_phase;"
-    attempt|execution|3
-    insight|execution|3
+    attempt|execution|5
+    distillation|cross_task_distill|1
+    distillation|per_task_distill|3
+    insight|execution|5
+    post|cross_task_forum|1
     ```
 
+    The `distillation` rows (one per-task bundle per generation, plus a cross-task
+    bundle) confirm the distill phase ran each generation, and
+
+    ```console
+    $ sqlite3 runtime_state/knowledge/quickstart_demo/quickstart_demo_knowledge.sqlite \
+        "select count(*) from seed_snapshots;"
+    2
+    ```
+
+    the two `seed_snapshots` confirm seeding fired between each pair of
+    generations. Exact counts vary with the model and how much each agent posts.
+
 ## What just happened?
 
 KSI runs a knowledge-refinement loop across generations:
@@ -106,14 +139,18 @@ KSI runs a knowledge-refinement loop across generations:
 4. The system [*distills*](glossary.md#distillation) those discussions into reusable guidance.
 5. The next generation is [*seeded*](glossary.md#seeding) with that guidance.
 
-!!! note "Why the demo doesn't show steps 3–5"
-    The quickstart runs a single generation with both forums off, so steps 3–5
-    don't fire here. And because every task solves on the first attempt, there
-    would be nothing to learn anyway: a solved task is dropped from later
-    generations (`--drop-solved`, on by default), so a multi-generation run
-    **stops early** once everything is solved. To watch the full loop, turn the
-    forums on, request several generations, and use tasks hard enough that some
-    fail — see [experiments.md](experiments.md).
+!!! note "Why ARC tasks — and why hard ones"
+    The demo uses ARC-AGI-1 tasks because they're hard for every current model:
+    if the tasks were easy, every agent would solve them on the first attempt and
+    — with `--drop-solved` (on by default) — the task pool would empty after
+    generation 1, so the run would **stop** before the forum, distill, and seed
+    phases could show their value. The three bundled tasks
+    (`97239e3d`, `d22278a0`, `776ffc46`) are chosen for low observed pass rates
+    (see [`examples/quickstart/arc1_hard/`](https://github.com/recursive-knowledge/KSI/tree/main/examples/quickstart/arc1_hard)),
+    so unsolved tasks carry forward under the default `--drop-solved` and the full
+    loop runs across all three generations. On your own *easy* tasks, expect the
+    run to stop early once everything is solved; that's the intended behavior. See
+    [experiments.md](experiments.md).
 
 ## Next steps