diff --git a/README.md b/README.md index 5a94517..7b9f85f 100644 --- a/README.md +++ b/README.md @@ -41,10 +41,12 @@ bash scripts/quickstart.sh The script self-bootstraps everything it needs: it synthesizes a provider profile from your key, builds the `ksi-agent:bench` image on first run, -installs the host Node dependencies, then runs one generation over the -bundled [`examples/custom_tasks/`](./examples/custom_tasks/) demo. If -anything is missing, `uv run ksi-doctor` prints a ✓/✗ readiness checklist -with the exact command to fix it. +installs the host Node dependencies, then runs three generations over three +hard [ARC-AGI-1](./examples/quickstart/arc1_hard/) tasks (bundled, no download) +— with the forums on, so the full execute → forum → distill → seed loop fires. +ARC tasks are hard for every current model, so they don't all solve on the first +generation and the loop keeps going. If anything is missing, `uv run ksi-doctor` +prints a ✓/✗ readiness checklist with the exact command to fix it. ## Documentation diff --git a/docs/experiments.md b/docs/experiments.md index 8f05ad0..6c1057d 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -1,9 +1,9 @@ # Running larger runs The [quickstart](getting-started.md) and [your own tasks](your_own_tasks.md) -walkthroughs run one generation over a handful of tiny tasks. This page -covers the flags that matter once you scale up — more tasks, more -generations, or a maintained reference benchmark instead of your own tasks. +walkthroughs run over a handful of tiny tasks. This page covers the flags +that matter once you scale up — more tasks, more generations, or a +maintained reference benchmark instead of your own tasks. There is **one canonical launch surface**: the `ksi.cli` argument parser. Everything else (bash presets, `ksi.run(...)`) is a layer over the same diff --git a/docs/faq.md b/docs/faq.md index 51cda64..c3cbece 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -83,8 +83,9 @@ Copy the template for your provider, fill in your key, and pass the path with Every run makes real LLM API calls billed to the key in your provider profile; there is no built-in spending cap. Cost scales with the number of tasks, generations, and the model you choose. To get a feel before committing, -start with the bundled synthetic demo: `bash scripts/quickstart.sh` runs -three tasks, one generation, with Haiku — the fastest and cheapest option. +start with the bundled demo: `bash scripts/quickstart.sh` runs three hard +ARC-AGI-1 tasks across three generations with Haiku — the cheapest way to +watch the full loop. Use `DRY_RUN=true` on any experiment wrapper script to print the full CLI command and DB paths without launching anything. diff --git a/docs/getting-started.md b/docs/getting-started.md index 7e167f2..8287857 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,12 +1,15 @@ # Getting started -Go from a fresh clone to a solved demo task in one command, then learn what just happened. +Go from a fresh clone to the full knowledge loop running in one command, then learn what just happened. ## What you'll do -Run one fast generation of agents against three bundled, self-contained tasks and see them score — -no dataset download, no manual setup. The whole demo finishes in a few minutes and leaves you with -a working environment ready to run your own tasks or a reference benchmark. +Run three generations of agents against five bundled **ARC-AGI-1** tasks — hard enough for any +current model that they don't all solve on the first try — and watch the full knowledge loop — +execute, discuss, distill, seed — fire end to end. No dataset download, no manual setup (the ARC-1 +tasks are vendored under `benchmarks/arc1/`). ARC attempts are slow, so a full three-generation +run takes on the order of 15–20 minutes; it leaves you with a working environment ready to run your +own tasks or a reference benchmark. ## Prerequisites @@ -24,10 +27,14 @@ bash scripts/quickstart.sh The script self-bootstraps everything it needs: it synthesizes a provider profile from your key, builds the `ksi-agent:bench` image on first run (this takes a few minutes), installs the host -Node dependencies, then runs one generation over the three bundled tasks under -[`examples/custom_tasks/`](https://github.com/recursive-knowledge/KSI/tree/main/examples/custom_tasks) -(`fizzbuzz`, `reverse-words`, `anagram-groups`) — each graded by running `python3 tests.py` -against the agent's attempt. +Node dependencies, then runs three generations over three hard **ARC-AGI-1** tasks — bundled under +[`examples/quickstart/arc1_hard/`](https://github.com/recursive-knowledge/KSI/tree/main/examples/quickstart/arc1_hard) +(copied from the ARC-1 corpus vendored under `benchmarks/arc1/`, no download) — with the per-task +and cross-task forums on so every phase of the loop fires. Each agent studies an ARC task's +input→output training examples and writes its predicted grids, scored by the `arc_session` +evaluator (exact-match, up to two attempts per test). These three tasks are chosen for being hard +for current models (see the [directory README](https://github.com/recursive-knowledge/KSI/blob/main/examples/quickstart/arc1_hard/README.md) +for their observed pass rates), so they don't all solve on the first generation. For the complete benchmark environment (including benchmark preparation and smoke tests), run `bash scripts/setup_all.sh`. Use `--no-test` when you need @@ -60,10 +67,16 @@ The run logs each attempt and its score as it progresses. When it finishes, resu | Score summary (optional — only when `--output-json` is set) | `results/.json` | | Execution traces | `analysis/traces//` | -For the quickstart, `` defaults to `quickstart_demo`. The run prints -each task's score as it goes and ends with a `completed … solved=3/3 (100.0%)` -line — that's the signal your environment is set up correctly. Elapsed times and -token counts vary by model and run; the task names and `solved=3/3` don't. +For the quickstart, `` defaults to `quickstart_demo`. The run logs +each task's score (ARC scoring is exact-match: `1.0` solved, `0.0` not) as it +goes, and ends with a single `completed traces=… tasks=… solved=N/M` summary +line. These ARC tasks are hard for current models, so expect at least one to +remain unsolved after generation 1 (a strong model may solve the rest); the +unsolved tasks carry forward and get re-attempted each generation, now seeded +with what the population distilled. **The signal that your environment is set up +correctly is that attempts run and get scored at all** — not that everything +solves. Whether an unsolved task flips to solved by generation 3 depends on the +model. Elapsed times, token counts, and solve counts vary by model and run. ??? note "A closer look — sample output, optional artifacts, and the knowledge DB" @@ -72,30 +85,50 @@ token counts vary by model and run; the task names and `solved=3/3` don't. run preset, which sets it for you) for a score summary on disk. Traces default to `analysis/traces//` — set `KSI_TRACE_DIR` to change the root. - A real excerpt from a run against `claude-haiku-4-5-20251001` (the default - `configs/ksi/.env.haiku` profile), timestamps trimmed: + An excerpt from a real run (`gpt-5.4-mini`, timestamps trimmed). ARC scores + are binary (exact-match); a strong model solves some tasks on generation 1 + while the hardest carry forward: ```text - INFO ksi.orchestrator.execution_phase: [gen 1] task=reverse-words agent=agent-1 done elapsed=27.4s score=1.0000 - INFO ksi.orchestrator.execution_phase: [gen 1] task=fizzbuzz agent=agent-0 done elapsed=28.1s score=1.0000 - INFO ksi.orchestrator.execution_phase: [gen 1] task=anagram-groups agent=agent-2 done elapsed=33.0s score=1.0000 - INFO ksi.orchestrator.engine: completed traces=3 tasks=3 solved=3/3 (100.0%) - INFO ksi.orchestrator.persistence: [tokens] total=418,329 cached_input=346,149 uncached_input=9,129 output=5,090 cache_create=57,961 + INFO ksi.orchestrator.execution_phase: [gen 1] task=776ffc46 agent=agent-0 done elapsed=281.6s score=1.0000 + INFO ksi.orchestrator.execution_phase: [gen 1] task=97239e3d agent=agent-1 done elapsed=281.6s score=1.0000 + INFO ksi.orchestrator.execution_phase: [gen 1] task=d22278a0 agent=agent-2 done elapsed=287.4s score=0.0000 + INFO ksi.orchestrator.distillation_phase: [ENGINE] distill gen=1: 1 per-task bundle(s), cross_task=0 + INFO ksi.orchestrator.persistence: [gen 2] start agents=1 ``` - **Knowledge DB check** — every solved attempt writes an `entry_type='attempt'` - row plus an `insight` row. The quickstart turns both forums off for speed - (`--per-task-forum-rounds 0 --cross-task-forum-rounds 0`), so there are no - discussion posts, and with nothing unsolved in this single-generation run - distillation has nothing to write either: + Here two tasks solved and dropped out (`--drop-solved`), while `d22278a0` + stayed unsolved and carried through generations 2 and 3 — each time + re-attempted with freshly distilled guidance seeded in. The run ended with + `completed traces=5 tasks=3 solved=2/3 (66.7%)`: five attempts across three + generations over three unique tasks. + + **Knowledge DB check** — because the demo runs the full loop, the knowledge + DB carries rows from every phase, not just execution. Group by `entry_type` + and `source_phase` to see them (real counts from the run above): ```console $ sqlite3 runtime_state/knowledge/quickstart_demo/quickstart_demo_knowledge.sqlite \ "select entry_type, source_phase, count(*) from knowledge group by entry_type, source_phase order by entry_type, source_phase;" - attempt|execution|3 - insight|execution|3 + attempt|execution|5 + distillation|cross_task_distill|1 + distillation|per_task_distill|3 + insight|execution|5 + post|cross_task_forum|1 ``` + The `distillation` rows (one per-task bundle per generation, plus a cross-task + bundle) confirm the distill phase ran each generation, and + + ```console + $ sqlite3 runtime_state/knowledge/quickstart_demo/quickstart_demo_knowledge.sqlite \ + "select count(*) from seed_snapshots;" + 2 + ``` + + the two `seed_snapshots` confirm seeding fired between each pair of + generations. Exact counts vary with the model and how much each agent posts. + ## What just happened? KSI runs a knowledge-refinement loop across generations: @@ -106,14 +139,18 @@ KSI runs a knowledge-refinement loop across generations: 4. The system [*distills*](glossary.md#distillation) those discussions into reusable guidance. 5. The next generation is [*seeded*](glossary.md#seeding) with that guidance. -!!! note "Why the demo doesn't show steps 3–5" - The quickstart runs a single generation with both forums off, so steps 3–5 - don't fire here. And because every task solves on the first attempt, there - would be nothing to learn anyway: a solved task is dropped from later - generations (`--drop-solved`, on by default), so a multi-generation run - **stops early** once everything is solved. To watch the full loop, turn the - forums on, request several generations, and use tasks hard enough that some - fail — see [experiments.md](experiments.md). +!!! note "Why ARC tasks — and why hard ones" + The demo uses ARC-AGI-1 tasks because they're hard for every current model: + if the tasks were easy, every agent would solve them on the first attempt and + — with `--drop-solved` (on by default) — the task pool would empty after + generation 1, so the run would **stop** before the forum, distill, and seed + phases could show their value. The three bundled tasks + (`97239e3d`, `d22278a0`, `776ffc46`) are chosen for low observed pass rates + (see [`examples/quickstart/arc1_hard/`](https://github.com/recursive-knowledge/KSI/tree/main/examples/quickstart/arc1_hard)), + so unsolved tasks carry forward under the default `--drop-solved` and the full + loop runs across all three generations. On your own *easy* tasks, expect the + run to stop early once everything is solved; that's the intended behavior. See + [experiments.md](experiments.md). ## Next steps diff --git a/examples/quickstart/arc1_hard/776ffc46.json b/examples/quickstart/arc1_hard/776ffc46.json new file mode 100644 index 0000000..c3befd0 --- /dev/null +++ b/examples/quickstart/arc1_hard/776ffc46.json @@ -0,0 +1 @@ +{"train": [{"input": [[5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [5, 0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [5, 0, 2, 2, 2, 0, 5, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [5, 0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [5, 0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [5, 0, 2, 2, 2, 0, 5, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [5, 0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 3, 3, 3, 0, 0], [0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5], [0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5], [0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 2, 2, 2, 0, 5], [0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 2, 2, 2, 0, 5], [0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5], [5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5], [0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5], [0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 2, 2, 2, 0, 5], [0, 0, 2, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 2, 2, 2, 0, 5], [0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5], [5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 2, 2, 2, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 2, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 3, 3, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 3, 3, 0, 0], [0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 3, 0], [0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0], [0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5], [0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0], [0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 3, 3, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 3, 3, 0, 0], [0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 5, 0, 3, 3, 3, 3, 0], [0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0], [0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5], [0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 5, 0, 3, 3, 3, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0], [0, 5, 5, 5, 5, 5, 5, 5, 0, 0, 3, 3, 3, 0, 0, 0, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 0, 0, 1, 1, 1, 1, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}], "test": [{"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 2, 2, 2, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 5, 0, 0, 2, 2, 2, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 5], [0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0], [0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0], [0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0], [0, 0, 2, 2, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 2, 0, 0, 5, 0], [0, 2, 2, 2, 2, 0, 5, 0, 0, 0, 0, 0, 5, 0, 2, 2, 2, 0, 5, 0], [0, 2, 2, 2, 2, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 2, 0, 0, 5, 0], [0, 0, 2, 2, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0], [0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0]], "output": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 2, 2, 2, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 5, 0, 0, 2, 2, 2, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 5], [0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 2, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0], [0, 2, 2, 2, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 2, 2, 2, 0, 0, 0], [0, 0, 2, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [5, 5, 5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0], [0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0], [0, 0, 2, 2, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 2, 0, 0, 5, 0], [0, 2, 2, 2, 2, 0, 5, 0, 0, 0, 0, 0, 5, 0, 2, 2, 2, 0, 5, 0], [0, 2, 2, 2, 2, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 2, 0, 0, 5, 0], [0, 0, 2, 2, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 0], [0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 5, 5, 5, 5, 5, 5, 5, 0]]}]} \ No newline at end of file diff --git a/examples/quickstart/arc1_hard/97239e3d.json b/examples/quickstart/arc1_hard/97239e3d.json new file mode 100644 index 0000000..6513de7 --- /dev/null +++ b/examples/quickstart/arc1_hard/97239e3d.json @@ -0,0 +1 @@ +{"train": [{"input": [[2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0], [2, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [2, 8, 2, 8, 0, 8, 2, 8, 0, 8, 2, 8, 2, 8, 0, 8, 0], [2, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0], [2, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [2, 8, 2, 8, 0, 8, 2, 8, 0, 8, 2, 8, 2, 8, 0, 8, 0], [2, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 6, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 1, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6], [0, 8, 8, 8, 6, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 6], [0, 8, 0, 8, 6, 8, 6, 8, 0, 8, 6, 8, 0, 8, 6, 8, 6], [0, 8, 8, 8, 6, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 6], [0, 0, 0, 0, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 8, 8, 8, 0, 8, 8, 8, 1, 8, 8, 8, 0, 8, 8, 8, 0], [1, 8, 1, 8, 0, 8, 1, 8, 1, 8, 0, 8, 0, 8, 0, 8, 0], [1, 8, 8, 8, 0, 8, 8, 8, 1, 8, 8, 8, 0, 8, 8, 8, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 7], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 7, 7, 7, 7], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 7, 8, 8, 8, 7], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 7, 8, 7, 8, 7], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 7, 8, 8, 8, 7], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 7, 7, 7, 7], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0], [3, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 3, 8, 8, 8, 0], [3, 8, 3, 8, 0, 8, 3, 8, 0, 8, 3, 8, 3, 8, 0, 8, 0], [3, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 3, 8, 8, 8, 0], [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}], "test": [{"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 4, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 4, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 2, 8, 0, 8, 2, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 2, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0], [0, 8, 8, 8, 4, 8, 8, 8, 0, 8, 8, 8, 4, 8, 8, 8, 0], [0, 8, 0, 8, 4, 8, 4, 8, 0, 8, 4, 8, 4, 8, 0, 8, 0], [0, 8, 8, 8, 4, 8, 8, 8, 0, 8, 8, 8, 4, 8, 8, 8, 0], [0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0], [0, 8, 8, 8, 4, 8, 8, 8, 0, 8, 8, 8, 4, 8, 8, 8, 0], [0, 8, 0, 8, 4, 8, 4, 8, 0, 8, 4, 8, 4, 8, 0, 8, 0], [0, 8, 8, 8, 4, 8, 8, 8, 0, 8, 8, 8, 4, 8, 8, 8, 0], [0, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0], [0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0, 8, 8, 8, 0], [0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0], [0, 8, 8, 8, 2, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [0, 8, 0, 8, 2, 8, 2, 8, 0, 8, 2, 8, 2, 8, 0, 8, 0], [0, 8, 8, 8, 2, 8, 8, 8, 0, 8, 8, 8, 2, 8, 8, 8, 0], [0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0]]}]} \ No newline at end of file diff --git a/examples/quickstart/arc1_hard/README.md b/examples/quickstart/arc1_hard/README.md new file mode 100644 index 0000000..510f942 --- /dev/null +++ b/examples/quickstart/arc1_hard/README.md @@ -0,0 +1,28 @@ +# Quickstart hard ARC-AGI-1 tasks + +Three ARC-AGI-1 tasks used by `scripts/quickstart.sh` to demonstrate the full +knowledge-refinement loop (execute → forum → distill → seed) across generations. + +They are chosen to be **hard for current models** — none is reliably one-shot, +so with `--drop-solved` (on by default) the unsolved tasks carry forward and the +forum/distill/seed phases actually run across all three generations, instead of +the pool emptying after generation 1. + +| task | ARC-1 split | observed pass rate (single attempt) | +|------|-------------|-------------------------------------| +| `97239e3d` | evaluation | GPT ~0/22, Haiku ~4/14 | +| `d22278a0` | training | GPT ~0/4, Haiku ~1/6 | +| `776ffc46` | training | GPT ~0/4, Haiku ~1/6 | + +Pass rates are approximate, from internal runs; they show these tasks are +solvable *sometimes* (so the knowledge loop has something to learn and transfer) +but rarely on the first try. + +## Provenance + +The task JSONs are copied verbatim from the vendored ARC-AGI-1 corpus under +`benchmarks/arc1/source/data/{evaluation,training}/`, originally from +[`fchollet/ARC-AGI`](https://github.com/fchollet/ARC-AGI) (Apache-2.0). They are +duplicated here only so the quickstart can point at a single `--tasks-path` +directory holding all three (they span two splits), keeping the demo +self-contained with no dataset download. diff --git a/examples/quickstart/arc1_hard/d22278a0.json b/examples/quickstart/arc1_hard/d22278a0.json new file mode 100644 index 0000000..bc84f5b --- /dev/null +++ b/examples/quickstart/arc1_hard/d22278a0.json @@ -0,0 +1 @@ +{"train": [{"input": [[1, 0, 0, 0, 0, 0, 0, 0, 0, 2], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[1, 0, 1, 0, 1, 2, 0, 2, 0, 2], [0, 0, 1, 0, 1, 2, 0, 2, 0, 0], [1, 1, 1, 0, 1, 2, 0, 2, 2, 2], [0, 0, 0, 0, 1, 2, 0, 0, 0, 0], [1, 1, 1, 1, 1, 2, 2, 2, 2, 2], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 2, 2, 2, 2, 2], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 2, 2, 2, 2, 2], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}, {"input": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[0, 3, 0, 3, 0, 3, 0, 3, 0, 3, 0, 3], [8, 0, 0, 3, 0, 3, 0, 3, 0, 3, 0, 0], [0, 0, 0, 3, 0, 3, 0, 3, 0, 3, 3, 3], [8, 8, 8, 0, 0, 3, 0, 3, 0, 0, 0, 0], [0, 0, 0, 0, 0, 3, 0, 3, 3, 3, 3, 3], [8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3, 3], [8, 8, 8, 8, 8, 0, 8, 0, 0, 0, 0, 0], [0, 0, 0, 0, 8, 0, 8, 0, 0, 3, 3, 3], [8, 8, 8, 0, 8, 0, 8, 0, 8, 0, 0, 0], [0, 0, 8, 0, 8, 0, 8, 0, 8, 0, 0, 3], [8, 0, 8, 0, 8, 0, 8, 0, 8, 0, 8, 0]]}, {"input": [[2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "output": [[2, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2], [0, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2], [2, 2, 2, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2], [0, 0, 0, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2], [2, 2, 2, 2, 2, 0, 2, 0, 2, 0, 2, 0, 2], [0, 0, 0, 0, 0, 0, 2, 0, 2, 0, 2, 0, 2], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 4, 0, 4, 0, 4, 0, 4], [4, 4, 4, 4, 4, 0, 4, 0, 4, 0, 4, 0, 4], [0, 0, 0, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4], [4, 4, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4], [0, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4], [4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4]]}, {"input": [[1, 0, 0, 0, 0, 0, 2], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0], [8, 0, 0, 0, 0, 0, 0]], "output": [[1, 0, 1, 0, 2, 0, 2], [0, 0, 1, 0, 2, 0, 0], [1, 1, 1, 0, 2, 2, 2], [0, 0, 0, 0, 0, 0, 0], [8, 8, 8, 0, 0, 2, 2], [0, 0, 8, 0, 8, 0, 0], [8, 0, 8, 0, 8, 0, 0]]}], "test": [{"input": [[4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]], "output": [[4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 0], [0, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 0], [4, 4, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 0, 1, 1], [0, 0, 0, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 0, 0, 0], [4, 4, 4, 4, 4, 0, 4, 0, 4, 0, 4, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 4, 0, 4, 0, 4, 0, 0, 0, 0, 0, 0], [4, 4, 4, 4, 4, 4, 4, 0, 4, 0, 0, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [8, 8, 8, 8, 8, 8, 8, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], [8, 8, 8, 8, 8, 0, 8, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1], [0, 0, 0, 0, 8, 0, 8, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0], [8, 8, 8, 0, 8, 0, 8, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1], [0, 0, 8, 0, 8, 0, 8, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0], [8, 0, 8, 0, 8, 0, 8, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1]]}]} diff --git a/scripts/quickstart.sh b/scripts/quickstart.sh index ae78d7e..0da8d9e 100755 --- a/scripts/quickstart.sh +++ b/scripts/quickstart.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# One command, fresh clone -> solved task on the bundled custom-tasks demo. +# One command, fresh clone -> the full knowledge loop on 5 hard ARC-AGI-1 tasks. # # No external dataset download and no prior setup_all.sh run needed. Given # Docker and Node.js 22.16.0 installed (and either uv or a local @@ -10,7 +10,11 @@ # from the environment); # - the ksi-agent:bench Docker image (built on first run if missing); # - the host runtime_runner Node dependencies. -# Then it runs a single minimal generation so the demo finishes in a few minutes. +# Then it runs 3 generations with the forums on over 3 hard ARC-AGI-1 tasks +# (bundled under examples/quickstart/arc1_hard/, no download). ARC tasks are hard +# for every current model, so they don't all solve on generation 1 — unsolved +# tasks carry forward under the default --drop-solved and the full knowledge loop +# (execute -> forum -> distill -> seed) fires end to end across generations. # # Usage: # ANTHROPIC_API_KEY=sk-ant-... bash scripts/quickstart.sh @@ -18,7 +22,10 @@ # PROFILE=configs/ksi/.env.openai bash scripts/quickstart.sh # # Env knobs: -# TASKS_PATH= run your own tasks .jsonl/.json instead of the bundled demo +# TASKS_PATH= run your own custom tasks .jsonl/.json (command evaluator) +# instead of the default ARC-AGI-1 demo +# ARC_DATA_DIR= directory of ARC task JSONs (default: the bundled 3 tasks) +# ARC_TASK_MAP=

optional ARC task-map JSON to filter/pin the selection # EXPERIMENT_NAME=x name the run (default: quickstart_demo) # PROFILE= provider profile to use (default: configs/ksi/.env.haiku) # SKIP_BOOTSTRAP=1 don't build the image / install deps / synthesize a profile @@ -33,10 +40,16 @@ KSI_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" cd "$KSI_ROOT" PROFILE="${PROFILE:-configs/ksi/.env.haiku}" -TASKS_PATH="${TASKS_PATH:-examples/custom_tasks/tasks.jsonl}" EXPERIMENT_NAME="${EXPERIMENT_NAME:-quickstart_demo}" AGENT_IMAGE="ksi-agent:bench" +# Demo task selection. Default: 3 hard ARC-AGI-1 tasks bundled under +# examples/quickstart/arc1_hard/ (no download). Set TASKS_PATH to a custom +# .jsonl/.json to run your own tasks through the command evaluator instead. +TASKS_PATH="${TASKS_PATH:-}" +ARC_DATA_DIR="${ARC_DATA_DIR:-examples/quickstart/arc1_hard}" +ARC_TASK_MAP="${ARC_TASK_MAP:-}" + # Prefer uv, but fall back to a plain interpreter so a `pip install`d package # (no uv) still works. uv is a convenience here, not a hard requirement. if command -v uv >/dev/null 2>&1; then @@ -130,22 +143,40 @@ if [[ ! -f "$PROFILE" ]]; then exit 1 fi -# Minimal canonical run: only the required flags plus one fast generation with -# the discussion/distillation phases off, so the demo finishes quickly. +# Full-loop demo: 3 generations with the per-task and cross-task forums on. The +# default task set is 3 hard ARC-AGI-1 tasks — hard enough for any current model +# that they don't all solve on generation 1, so unsolved tasks carry forward +# under the default --drop-solved and every phase fires across generations +# (execute -> forum -> distill -> seed -> next generation). Setting TASKS_PATH +# switches to a custom .jsonl/.json graded by the command evaluator. +if [[ -n "$TASKS_PATH" ]]; then + TASK_FLAGS=(--task-source custom --tasks-path "$TASKS_PATH" --evaluator command) + TASK_BANNER="custom tasks from $TASKS_PATH" + CONCURRENCY=4 +else + TASK_FLAGS=( + --task-source arc + --tasks-path "$ARC_DATA_DIR" + --evaluator arc_session + --arc-max-trials 2 + ) + [[ -n "$ARC_TASK_MAP" ]] && TASK_FLAGS+=(--task-map-path "$ARC_TASK_MAP") + TASK_BANNER="3 hard ARC-AGI-1 tasks" + CONCURRENCY=3 +fi + CMD=( "${PYRUN[@]}" -m ksi.cli - --task-source custom - --tasks-path "$TASKS_PATH" - --evaluator command + "${TASK_FLAGS[@]}" --provider-profile "$PROFILE" - --generations 1 - --per-task-forum-rounds 0 - --cross-task-forum-rounds 0 - --max-concurrent-tasks 3 + --generations 3 + --per-task-forum-rounds 1 + --cross-task-forum-rounds 1 + --max-concurrent-tasks "$CONCURRENCY" --experiment-name "$EXPERIMENT_NAME" ) -echo "==> Running quickstart demo (3 custom Python tasks: fizzbuzz, reverse-words, anagram-groups)" +echo "==> Running quickstart demo ($TASK_BANNER)" echo " ${CMD[*]}" if [[ "${DRY_RUN:-false}" == "true" ]]; then