Repository navigation
feat(catalog): read trigger sets back and list skill package formats - #211
Merged
Merged
Conversation
A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi catalog formats`
+ option `--help/-h`
+ option `--json`
+ option `--root`
+ added command `dokimi tasks inspect`
+ argument `path`
+ option `--help/-h`
+ option `--json`
~ `dokimi catalog trigger-set` parameters changed:
+ added option `--trigger-set`
~ changed option `--questions`: required: True -> FalseUpdated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #211 +/- ##
==========================================
- Coverage 92.09% 92.07% -0.02%
==========================================
Files 225 233 +8
Lines 28101 29090 +989
Branches 4622 4787 +165
==========================================
+ Hits 25879 26784 +905
- Misses 1576 1620 +44
- Partials 646 686 +40
🚀 New features to boost your workflow:
|
napetrov
force-pushed
the
feat/run-trigger-formats
branch
from
October 24, 2026 17:13
29df5da to
f7abf7c
Compare
Readers for the three trigger-set formats dokimi already writes (skill-reach QuerySet, skill-creator / agentskills.io eval queries, questions with expected_skill) into TriggerQuery, held to the writers by round-trip tests. catalog trigger-set --trigger-set relints an imported set against the catalog and hands it to the skill-catalog: arm; arms run points external sets at it. catalog formats names the skill/rule packages of other ecosystems (Cursor rules, Copilot instructions, Gemini extensions, Claude plugins) next to agentskills.io SKILL.md, with spec lint and the selecting mechanism. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Hits over the questions where the arm loaded any skill, strict on multi-load like recall; rendered in the arms report's Skill selection table. Replaces the ad-hoc calculation used for the intel#211 evidence table. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
This was referenced Oct 24, 2026
napetrov
marked this pull request as ready for review
October 25, 2026 00:36
napetrov
requested review from
antonwolfy,
jharlow-intel,
vchamarthi and
xaleryb
as code owners
October 25, 2026 00:36
build_trigger_set rebuilt each query from its text alone, so a questions set read back with --trigger-set (or --questions) lost its pre-rewrite source_text. Carry it, and keep it as the original on a second --rewrite. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
`catalog formats` aborted the whole scan with a TypeError when a gemini-extension.json had a number or boolean mcpServers, and counted a string by its characters. Record it as a package problem instead. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
…d back Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Pushed 98cb170 for the first point above: a reach set written back as reach keeps each entry's Not covered: reach -> questions -> reach still drops them, since keeping them would mean declaring |
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Pushed 21b6433 for the The duplicate-id error across repeated |
xaleryb
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 25, 2026
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> intel#211 landed with a copy of this branch's first commit (f4ce883) plus its own reach consumers. Each conflicting file was merged three-way against f4ce883, so main's additions (BY_TRIGGER_SET and its consumers) and this branch's later commits are both kept.
xaleryb
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 25, 2026
…ntel#211 Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
xaleryb
added a commit
that referenced
this pull request
Oct 26, 2026
* feat(tasks): detect eval-task formats with `tasks inspect` A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): task_inspect.v1 report, config-named datasets, LFS pointers `tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(tasks): real-format corpus for tasks inspect, with its manifest tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(tasks): keep a tree scan alive on NUL paths and non-string YAML keys A config naming a path with a NUL byte raised ValueError in links._key, and a skill-reach key YAML loads as bool or int (on:, yes:, 1:) raised TypeError in the key lint; either one killed the whole `tasks inspect`. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): write YAML dates, sets and NaN in details as plain JSON Probes copy YAML values into details unchanged. --json hid it with default=str, but --out (save_artifact) raised TypeError and wrote nothing, and a YAML .nan went out as bare NaN in both modes. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): do not read FIFOs in a tree scan; cap reads of sizeless files The 64 MiB guard trusted st_size, which is 0 for a FIFO or device: a FIFO left in a scanned tree hung the scan, and /dev/zero was read until memory ran out. The scan now takes regular files only and the read is bounded. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): claim docker-* only for tasks with environment/Dockerfile docker-* always builds environment/Dockerfile, so a harbor or task-md task without it errors in every cell, yet inspect listed docker-* as a consumer whenever tests/ or verifier/ existed. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): pin the guards that keep a tree scan finite Removing the symlink skip, the dot-directory skip, RecursionError from the parse catch or the file-cap flag left every task-format test green; two of those turn into crashes on real trees. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): report a file a tree scan could not read, as a single-file inspect does Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * ci(codefactor): leave the task-format corpus out; it mirrors upstream files Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * ci(codefactor): the exclusion key is exclude_patterns Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * ci(codefactor): drop .codefactor.yml; CodeFactor did not apply it on this PR Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): catalog trigger-set is a consumer of trigger sets since #211 Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): fix the three CodeFactor findings in the copied corpus files solution.sh: shebang above the canary line; Dockerfile: pip3 --no-cache-dir; verifier/test.sh: test pytest's exit code directly. Detection reads only which files exist, and MANIFEST.yaml notes each change, as its header asks. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): scan Inspect FieldSpec calls in linear time _FIELD_SPEC's [^)]* let each unclosed FieldSpec( run to the end of the segment, and search retried at every following one: 40000 of them after an @task file's dataset call took 43 s. The arguments are now bounded at 1000 characters and a run stops at the next FieldSpec(, so each start scans at most up to the next one. The bound alone (the review's [^)]{0,500}) is linear too but still took 17 s on a file at the 64 MiB read cap; with the stop it takes 1.2 s. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): read a CSV with a UTF-8 BOM without it in the first column An Excel or Sheets export of an Inspect samples.csv or a promptfoo tests.csv starts with a BOM, which csv.DictReader kept in the first header (\ufeffinput), so the file was unknown and a BOM on __expected hid the promptfoo marker. Both consumers drop it: Inspect's csv_dataset reads utf-8-sig and promptfoo parses CSV with bom: true. The BOM is stripped for CSV only, not by opening every file as utf-8-sig. A JSON or JSONL file with a BOM stays a 'not valid JSON: Unexpected UTF-8 BOM' problem, because that is what its readers do: read_json_input (catalog trigger-set --trigger-set), load_questions and Inspect's json_dataset all decode plain utf-8, so stripping it here would advertise a consumer that refuses the file. YAML needs nothing: the loader skips a leading BOM. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): report a task.md a tree scan cannot read instead of aborting TaskMdProbe.fingerprint reads task.md (_is_benchflow_task_md, _has_frontmatter) and walk called detect outside any try, so one mode-000 task.md ended inspect_path with a PermissionError and no report for any other task. walk now catches OSError around detect, the one place that covers every directory probe, and reports the directory as unknown with 'cannot read: ...', as an unreadable file in a tree is. This also covers a directory whose entries cannot be stat'ed (a mode-000 directory holding a task.toml), which failed the same way before this PR in task_format. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): pin the verifier condition of the docker-* claim _docker_runnable needs environment/Dockerfile and a tests/ or verifier/ directory, but only the Dockerfile half was tested: returning just the Dockerfile check passed every test. A Harbor and a task-md task with both verifier directories removed must not be claimed for docker-*. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): cover three guards the mutation run left untested - the samples of a model-graded OpenAI evals spec are read without the ideal lint (graded = False passed every test); - promptfoo tests: file://tests.csv labels a CSV that has no __expected column (the only fixture CSV is claimed with or without the reference); - a scan does not enter node_modules/ or venv/ (only .git and .venv were tested, and those are skipped as dot directories anyway). Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>
napetrov
added a commit
that referenced
this pull request
Oct 28, 2026
…217) * feat(tasks): detect eval-task formats with `tasks inspect` A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(catalog): read trigger sets back and list skill package formats Readers for the three trigger-set formats dokimi already writes (skill-reach QuerySet, skill-creator / agentskills.io eval queries, questions with expected_skill) into TriggerQuery, held to the writers by round-trip tests. catalog trigger-set --trigger-set relints an imported set against the catalog and hands it to the skill-catalog: arm; arms run points external sets at it. catalog formats names the skill/rule packages of other ecosystems (Cursor rules, Copilot instructions, Gemini extensions, Claude plugins) next to agentskills.io SKILL.md, with spec lint and the selecting mechanism. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(arms): precision in skill_selection_summary Hits over the questions where the arm loaded any skill, strict on multi-load like recall; rendered in the arms report's Skill selection table. Replaces the ad-hoc calculation used for the #211 evidence table. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(catalog): trigger-probe runs each query in real Claude Code; payload leak audit catalog trigger-probe runs every trigger-set query in a fresh claude -p: a new git repo with the catalog committed in .claude/skills, scratch HOME and config, the query verbatim, bundled skills on. Skill calls are scored by the same selection summary as the skill-catalog arm, one column per trial. payload_audit reports dokimi artifacts (arm names, label keys, question ids, grading words, task paths) in a captured request, argv or env, with operator content blanked and the stock harness's own text subtracted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(catalog): group trigger-probe Skill calls by message.id Claude Code streams one assistant event per content block, so two Skill calls in one turn were scored as two turns and the strict first-turn rule, multi_load_rate and mean_first_turn_loads never fired. Group by message.id as agent_runner groups by iteration, and skip subagent events. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(catalog): score a trigger-probe choice made before a timeout run_query checked timed_out first, so a query whose Skill call was already in the transcript, or whose result event was read before the CLI hung on a backgrounded Bash call, was dropped as n_failed. A Skill call seen is now the choice, and a result event decides the rest. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(catalog): apply the arms' plugin seal in trigger-probe --sealed --sealed set only the env seal, so the built-in plugin skills the docker-claude arms switch off via --settings stayed listed. Sealed argv now carries the same enabledPlugins, merged with --listing-budget-fraction into one --settings. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(catalog): keep trigger-probe results when one run fails A relative --claude passed the which() preflight but was spawned from the scratch work dir, and any OSError/RuntimeError in one run escaped pool.map and lost every paid result before --out was written. Resolve --claude to an absolute path and record a failed run as that trial's error. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(catalog): address trigger-probe review on #217 - tools_before_choice stops at the first Skill turn. - --out directory and --target are checked before any paid run; records are written after each query, so an interrupt keeps what ran. - Fail when a catalog skill is missing from the init event's listing; report runs that ended on error_max_turns (max_turns_hit). - Kill processes left in the run's workspace (a backgrounded Bash job runs in its own session and escaped the group kill). - Docs: the operator's config is not loaded, but host paths stay readable. - Tests: every CLI flag reaches ProbeConfig, argv/env/timeout/stdin of a run, symlinked skills copied as files, payload-audit case, lookbehind, tuple and longest-first blanking. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(catalog): trigger-probe self-review: partial file, input errors, result_subtype Keep in-progress records in <out>.partial so a rerun does not clobber a finished result; refuse a non-writable --out and a malformed catalog as input errors (exit 2) before any run; name the field result_subtype as the rest of the codebase does; document max_turns_hit, not_listed and the partial file. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Reject a non-positive --timeout before any run Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(catalog): pin trigger-probe's stray-process kill in run_query Only kill_strays itself was tested, so dropping its call from run_query or matching a sibling root's prefix left the suite green. Run a fake claude that leaves a job in its workspace and one in a same-prefix sibling root: the first must die, the second must survive. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(catalog): refuse a trigger-probe label that names no catalog skill A query whose expected_skill is not in --catalog can never be loaded, so it scored as a miss after the paid run: a typo read as recall 0. The skill-catalog: arm refuses such a label; trigger-probe now does too, before any run, as it already does for --target. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #210 (feat/task-format-detect). Review the commits after #210: f7abf7c, 19822ba (
precisioninskill_selection_summary). The fix for vendored-skill discovery is a separate PR, #215.Summary
dokimi could already write three trigger-set formats. With this PR it can read them too and run them through the existing
skill-catalog:arm. It adds no new runner.Evidence
arms run --questions reach.jsonfailed in the questions loader, andtasks inspectgave no consumer for skill-reach or skill-creator sets.After:
catalog trigger-set --trigger-set reach30.json --format questionsreads the file and re-lints it (30 queries: 19 exam-framed, 17 echo). Arms then score it. The round-trip tests (writer, then reader, gives the same labels) are intests/test_trigger_set_readers.py, and the package scan tests are intests/test_skill_formats.py. The full suite passes locally (7834), along with ruff and mypy.skill_selection_summarynow reportsprecision, which is hits over the questions where any skill loaded. The arms report shows it as a column. The precision figures below came from an ad-hoc script that used the same definition.skill-catalog:with reach30, 33 skillsskill-catalog:with onetbb skill-creator set, 33 skillsskill-catalog:with onetbb skill-creator set, target alonerun_eval.pynative (anthropics/skills@dbd4588), haikuThe tool-calling proxy triggers too often compared with the real harnesses. The import path is unaffected by this; it is the same finding as before.
Merge Danger
Door: two-way
Additive only: two new CLI options or subcommands and a new module.
--questionsoncatalog trigger-setis no longerrequired(at least one source is still enforced).Blast Radius: small
A skill-creator negative is read as "no skill should load". In a catalog of more than one skill, loading a rival skill therefore counts as a false trigger. The CLI warns about this.
🤖 Generated with Claude Code