feat(tasks): detect eval-task formats with tasks inspect

.github/.licenserc.yaml

@@ -59,6 +59,9 @@ header:
     # CHANGELOG.md, which carries the header itself.
     - 'dokimi/changelog.d/[0-9]*.md'
     - 'dokimi/changelog.d/fragment_template.md.j2'
+    # Third-party eval files copied unchanged, under their own licenses
+    # (dokimi/tests/fixtures/task_formats/MANIFEST.yaml, LICENSES/).
+    - 'dokimi/tests/fixtures/task_formats/**'
 
   comment: never
 

dokimi/changelog.d/20261023_120000_napetrov_task-format-detect.md

@@ -1,11 +1,16 @@
 ### Added
 
-- **`tasks inspect <path> [--json]`** — detects the eval-task format of a file,
-  a task directory or a whole tree and reports its structure: kind
-  (executable, answer-only, trigger), item count, present and missing parts,
-  lint problems and which dokimi backends consume it today. Recognises Harbor,
-  BenchFlow `task.md`, terminal-bench v1, SWE-bench instances, Inspect AI
-  samples and `@task` files (scanned as text, never imported), OpenAI evals
-  JSONL, skill-creator `evals.json` and trigger sets, skill-reach query sets,
-  promptfoo configs and dokimi question sets; anything else is reported as
-  `unknown` with its files. Detection only: no new format runs yet.
+- **`tasks inspect <path> [--json] [--out FILE]`** — detects the eval-task
+  format of a file, a task directory or a whole tree and reports its
+  structure: kind (executable, answer-only, trigger, localization), item
+  count, present and missing parts, lint problems and which dokimi backends
+  consume it today. Recognises Harbor, BenchFlow `task.md`, terminal-bench v1,
+  SWE-bench instances, Inspect AI samples (JSONL and CSV) and `@task` files
+  (scanned as text, never imported), OpenAI evals JSONL and registry specs,
+  skill-creator `evals.json` and trigger sets, skill-reach query sets,
+  promptfoo configs and test files, dokimi question sets and ExploreBench
+  tasks; anything else is reported as `unknown` with its files. A dataset
+  file a config names (an Inspect `FieldSpec` remap, an OpenAI evals
+  model-graded spec, promptfoo `tests: file://`) is read as that config says,
+  and a git-LFS pointer stub is reported as one. `--json` and `--out` write
+  the `task_inspect.v1` artifact. Detection only: no new format runs yet.

dokimi/docs/coding-harnesses.md

@@ -80,7 +80,7 @@ and times become ISO strings, and YAML sets sorted lists.
 
 ### Other eval formats: `tasks inspect`
 
-`python cli.py tasks inspect <path> [--json]` names the eval-task format of a
+`python cli.py tasks inspect <path> [--json] [--out FILE]` names the eval-task format of a
 file, a task directory, or every item under a tree, and reports its structure:
 kind, item count, which parts are present or missing (instruction,
 environment, oracle, verifier, metadata), lint problems, and what in dokimi
@@ -89,21 +89,38 @@ file is scanned as text and never imported, and a file that does not parse is
 a problem on its report, not an error. A path no format claims is reported as
 `unknown` with its files (or keys) and likely prompt and verifier files; when
 two formats claim a path about equally, the report lists the other as
-`ambiguous` with its evidence.
+`ambiguous` with its evidence. `--json` prints and `--out` writes the report
+as a `task_inspect.v1` artifact (`schemas/task_inspect.v1.json`).
+
+A dataset file a config in the same tree names is read as that config says:
+an Inspect `@task` file's `json_dataset("x.jsonl", FieldSpec(input="question",
+target="answer"))` makes `x.jsonl` Inspect samples under those keys, an OpenAI
+evals registry spec makes its `samples_jsonl` OpenAI evals (with no `ideal`
+needed when the spec's class is model-graded), and promptfoo `tests:
+file://tests.csv` makes that file promptfoo tests. The report says which file
+named it. A file that is a git-LFS pointer stub (a clone without `git lfs
+pull`) is reported as `git-lfs-pointer`, with the format a config expects it
+to be when one names it. `tests/fixtures/task_formats/` holds a small corpus of
+real files in each format, with their sources in its `MANIFEST.yaml`.
 
 | Format | Recognised by | Kind | Consumed by |
 |---|---|---|---|
-| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` |
-| `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` |
+| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` (with `environment/Dockerfile`) |
+| `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` (with `environment/Dockerfile`) |
 | `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | none yet |
 | `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | none yet |
-| `inspect-samples` | rows with `input` and `target` | answer-only | none yet |
+| `inspect-samples` | JSON/JSONL/CSV rows with `input` and `target` (or the keys a `FieldSpec` names) | answer-only | none yet |
 | `inspect-task` | `.py` importing `inspect_ai` with a `@task` function | answer-only (executable with a `sandbox=`) | none yet |
 | `openai-evals` | JSONL rows with `input` (chat messages) and `ideal` | answer-only | none yet |
+| `openai-evals-spec` | registry YAML with `class: evals.elsuite.*` run specs | answer-only | none yet |
 | `skill-creator-evals` | `evals.json` with `skill_name` and `evals[].prompt` | answer-only | none yet |
-| `skill-creator-trigger` | rows with `query` and `should_trigger` | trigger | none yet |
-| `skill-reach` | `QuerySet` (`queries[]` with `query_id`/`id`, `text`, `expected_skill`) | trigger | none yet |
+| `skill-creator-trigger` | rows with `query` and `should_trigger` | trigger | `catalog trigger-set` |
+| `skill-reach` | `QuerySet` (`queries[]` with `query_id`/`id`, `text`, `expected_skill`) | trigger | `catalog trigger-set` |
+| `promptfoo` | `promptfooconfig*.yaml`/`.json`, or a config with `prompts`, `providers`/`targets` and `tests`/`scenarios`/`redteam` | answer-only | none yet |
+| `promptfoo-tests` | CSV with `__expected` columns, or rows of `{vars, assert}` | answer-only | none yet |
 | `dokimi-questions` | questions.v1/v2 JSON | answer-only (trigger with `expected_skill`) | questions track |
+| `dokimi-explore-task` | YAML with `id`, `query` and `references[].path` | localization | explore track |
+| `git-lfs-pointer` | a git-LFS spec v1 pointer stub | unknown | none (fetch it first) |
 
 `tasks list` and `tasks run` still read only the first two.
 

dokimi/dokimi/artifacts.py

@@ -10,8 +10,9 @@
 model), ``guidance_reach`` (L1 reach probes), ``repo_readiness`` (L0 claim
 checks), ``tool_preference`` (which of two equal tools a model picks, and
 what moves it), ``repo_environment`` (static inspection of what a
-repository offers an agent) and ``repo_budget`` (what a repository puts
-into an agent's context). This module stamps each with a
+repository offers an agent), ``repo_budget`` (what a repository puts
+into an agent's context) and ``task_inspect`` (which eval-task format each
+path under a tree is). This module stamps each with a
 ``schema_version`` and validates it against the JSON Schema in
 ``dokimi/schemas/`` on save and load, so fixtures and baselines stay
 forward-compatible and self-describing.
@@ -52,6 +53,7 @@
     "tool_preference": "tool_preference.v1",
     "repo_environment": "repo_environment.v1",
     "repo_budget": "repo_budget.v1",
+    "task_inspect": "task_inspect.v1",
 }
 
 # kind -> superseded versions still accepted on *load*, newest first. An artifact written under an

dokimi/dokimi/commands/tasks.py

@@ -72,14 +72,21 @@ def cmd_tasks_inspect(args: argparse.Namespace) -> None:
     """Detect the eval-task format of a path (or of everything under it) and report its structure."""
     import json
 
+    from dokimi.artifacts import save_artifact, stamp
     from dokimi.task_formats import inspect_path
 
     try:
         reports = [r.as_dict() for r in inspect_path(Path(args.path))]
     except FileNotFoundError as exc:
         raise SystemExit(f"error: {exc}") from None
+    payload = {"root": str(args.path), "items": reports}
+    if args.out:
+        try:
+            save_artifact("task_inspect", payload, Path(args.out))
+        except OSError as exc:
+            raise SystemExit(f"error: cannot write {args.out}: {exc}") from None
     if args.json:
-        print(json.dumps({"root": str(args.path), "items": reports}, indent=2, default=str))
+        print(json.dumps(stamp("task_inspect", dict(payload)), indent=2, default=str))
     else:
         _print_inspect(reports)
 
@@ -227,7 +234,8 @@ def register(sub, positive_int) -> None:
         help="Detect the eval-task format of a file or directory tree and report its structure (no run)",
     )
     inspect_p.add_argument("path", help="A task directory, a dataset file, or a tree holding either")
-    inspect_p.add_argument("--json", action="store_true", help="Print the reports as JSON")
+    inspect_p.add_argument("--json", action="store_true", help="Print the task_inspect.v1 report as JSON")
+    inspect_p.add_argument("--out", default=None, help="Write the task_inspect.v1 report to this file")
     inspect_p.set_defaults(func=cmd_tasks_inspect)
 
     harnesses_p = tasks_sub.add_parser("harnesses", help="List known coding harness aliases")

dokimi/dokimi/schemas/task_inspect.v1.json

@@ -0,0 +1,65 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/intel/dokimi/schemas/task_inspect.v1.json",
+  "title": "Eval-task format inspection v1",
+  "description": "What `tasks inspect <path>` found under a path, from the files alone: nothing is imported, run or converted (dokimi/docs/coding-harnesses.md). One item per task directory or dataset file a format probe claims; a path nothing claims is a single `unknown` item. `details` is per format and open; every other level is closed.",
+  "type": "object",
+  "required": ["schema_version", "root", "items"],
+  "additionalProperties": false,
+  "properties": {
+    "schema_version": {"const": "task_inspect.v1"},
+    "root": {"type": "string", "description": "The path given to `tasks inspect`, as given."},
+    "items": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/item"}}
+  },
+  "$defs": {
+    "format": {
+      "enum": ["harbor", "task-md", "terminal-bench-v1", "swe-bench", "inspect-samples", "inspect-task",
+               "openai-evals", "openai-evals-spec", "skill-creator-evals", "skill-creator-trigger",
+               "skill-reach", "promptfoo", "promptfoo-tests", "dokimi-questions", "dokimi-explore-task",
+               "git-lfs-pointer", "unknown"],
+      "description": "The probe that claimed the item. `git-lfs-pointer` is a dataset file whose content was never fetched (`details.expected_format` names its format when a config in the scan names it); `unknown` is a path no probe claims."
+    },
+    "part": {"enum": ["instruction", "environment", "oracle", "verifier", "metadata"]},
+    "item": {
+      "type": "object",
+      "required": ["path", "format", "kind", "confidence", "evidence", "items", "present", "missing",
+                   "problems", "consumers", "alternatives", "details"],
+      "additionalProperties": false,
+      "properties": {
+        "path": {"type": "string"},
+        "format": {"$ref": "#/$defs/format"},
+        "kind": {
+          "enum": ["executable", "answer-only", "trigger", "localization", "unknown"],
+          "description": "`executable`: an environment and a verifier; `answer-only`: prompt/target samples; `trigger`: skill-selection queries; `localization`: where-is questions scored on cited locations."
+        },
+        "confidence": {"type": "number", "minimum": 0, "maximum": 1, "description": "0 for `unknown`."},
+        "evidence": {"type": "array", "items": {"type": "string"}},
+        "items": {"type": ["integer", "null"], "minimum": 0, "description": "Records in a dataset file; null for a task directory or an unread file."},
+        "present": {"type": "array", "items": {"$ref": "#/$defs/part"}},
+        "missing": {"type": "array", "items": {"$ref": "#/$defs/part"}},
+        "problems": {"type": "array", "items": {"type": "string"}, "description": "Lint findings: what a consumer would refuse or misread."},
+        "consumers": {
+          "type": "array",
+          "minItems": 1,
+          "items": {"enum": ["harbor", "docker-*", "benchflow", "questions track", "explore track", "catalog trigger-set", "none-yet"]},
+          "description": "What in dokimi reads the item today; `none-yet` when nothing does."
+        },
+        "alternatives": {
+          "type": "array",
+          "description": "Other probes that claimed the item about as strongly.",
+          "items": {
+            "type": "object",
+            "required": ["format", "confidence", "evidence"],
+            "additionalProperties": false,
+            "properties": {
+              "format": {"$ref": "#/$defs/format"},
+              "confidence": {"type": "number", "minimum": 0, "maximum": 1},
+              "evidence": {"type": "array", "items": {"type": "string"}}
+            }
+          }
+        },
+        "details": {"type": "object", "description": "Per-format structure (metadata summary, field map, references to dataset files, LFS oid and size)."}
+      }
+    }
+  }
+}