GitHub Action: pr-check / record / revalidate, .github/dokimi config, per-subject baselines #318

Goal

A repository that keeps skills (or agent guidance) can check every PR with dokimi on GitHub, keep a history of how each skill measured, and re-check old conclusions when a model changes. This needs no maintainer help and no verdict logic in YAML.

This is the Action part of #121 (item 5). It uses the CLI loop from #94 (§2.3, §2.6, §2.9) and serves journeys J2, J3, J4, J10, J12 and J13 (docs/user-journeys/journeys.md) and the catalog consumer #138.

Shape: a Marketplace action, sub-actions and reusable workflows

This follows the abicheck/abicheck layout. abicheck is listed on the Marketplace with a root action.yml and its own CI workflows in the same repo, so dokimi needs no separate action repo.

• actions/setup is public, the setup-python of dokimi. It installs dokimi pinned by SHA, sets the stable --out-dir layout, the cache and CI=true, and guards --max-usd on paid steps (action/setup: composite setup action for dokimi CI; repo-eval uses it (part of #318) #319).
• A composite action or reusable workflow cannot reach a sibling action by a relative path, because ./ resolves in the caller's repo. They refer to intel/dokimi/actions/<name> at the workflow's own SHA.
• The subject kind (skill | repo | catalog | docs) is an input. It is not a separate workflow. repo-eval.yml becomes dokimi-pr.yml with subject: repo and stays as a thin shortcut (the L6 pattern in CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94).
• The catalog (Skill catalog integration (intel/skills): per-skill evals in catalog CI, results, promotion to validated #138) is dokimi-pr.yml with plan catalog ROOT --changed-from BASE. Only the changed skills and their routing neighbours run.
• Workflows never compute a verdict. The job conclusion follows the exit-code table in CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 §2.6: 0 pass, 1 fail, 2 input error, 3 inconclusive, 4 invalid, 5 infra_error. Whether inconclusive blocks is decided by the policy.
• Workflow and action inputs use the flag names from cli_flags.py (§2.4) and are versioned as their own contract. Pin an exact tag or a SHA, and always a SHA when the job has write permissions (the abicheck rule).

How a user calls it

The main path is one file plus the config:

# .github/workflows/dokimi.yml
name: dokimi
on:
  pull_request:
  push: {branches: [main]}
  workflow_dispatch:
    inputs:
      plan: {type: choice, options: [smoke, full], default: smoke}
permissions: {}
jobs:
  dokimi:
    uses: intel/dokimi/.github/workflows/dokimi-pr.yml@<sha>   # v0.x
    permissions: {contents: read, pull-requests: write}
    with:
      plan: ${{ inputs.plan || 'smoke' }}
    secrets: inherit

• Add an environment dokimi-model with required reviewers. That is the per-SHA approval before any spend.
• Users who assemble their own jobs call the root action: uses: intel/dokimi@<sha> with mode: check or mode: run.
• dokimi assess --init writes this file, .github/dokimi/config.yaml, and with --forks also the publisher file.

PR comments

• A pull_request run from a fork gets a read-only token and no secrets, whatever permissions: says, so it cannot comment.
• Default: the job summary and the check status always, plus a sticky comment when the token allows it (same-repo branches).
• Optional, for repos with outside contributors: dokimi-publish.yml on workflow_run (the ci: post the cli-interface label and comment from a workflow_run job #151 pattern). The workflow_run trigger has to be declared in the caller's repo, because a reusable workflow has no triggers of its own. That is why it is a second, ~10-line caller file.

Credentials

• No secret is an input. The CLI reads ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, AWS credentials (Bedrock) and DOKIMI_GATEWAY_URL from the environment.
• With secrets: inherit, dokimi-pr.yml maps the known names into the env of the model-run job only. The caller names no key.
• The root composite action cannot read secrets, so its caller sets job env:. This is a GitHub limitation.
• OIDC (id-token: write, Bedrock, Azure or Vertex) is recommended where available, since then no long-lived key exists.
• A requested model run without credentials is an error, not a skip. mode: run, plan: smoke|full, a dispatch, or mode: auto that decided a model run is needed: validate fails before any install or spend with error.code: no_credentials, exit 2. The message names the providers it looked for.
• mode: check needs no credentials. A fork PR waiting for approval is pending at the environment. It is not "no credentials".
• not_run stays a cell-level status for declared reasons only: unsupported host, egress task, allow-unsupported.
• Logs print the names of the providers found, never the values.

Use cases

• A. A skill author edits SKILL.md (J2, J12). The free job finds the touched skill. If the frontmatter changed, it runs routing against the catalog. A maintainer approves the SHA, then the model job runs none, approved and candidate arms on the base suite. The PR gets an effect with its interval and a verdict.
• B. A repo maintainer edits AGENTS.md or build scripts (J3, J4). This is today's repo-eval.yml: the changed tasks, plus the smoke set when shared harness files change. The full run is a manual dispatch of dokimi-pr.yml with plan: full.
• C. Merge to main (J12). dokimi-record.yml promotes or re-runs, then advances the baseline of that subject only.
• D. A new model ships (J13). dokimi-revalidate.yml keeps the skill SHA fixed and varies only the model. The none arm always reruns. The result is reported separately from the effect of skill edits.
• E. A skills catalog (Skill catalog integration (intel/skills): per-skill evals in catalog CI, results, promotion to validated #138). A PR that touches 2 of 40 skills checks those 2 and their routing neighbours.
• F. A fork PR. The free layer runs at once. The model job runs only after a maintainer approves that exact SHA.

Security split (decided)

1. evaluate: pull_request, no secrets, read-only token. Free steps only. The grader, policy and config come from the base ref.
2. model-run: a GitHub environment that requires maintainer approval of a specific SHA. A new push invalidates the approval. This is the only job with secrets.
3. publish: workflow_run with comment permission, the ci: post the cli-interface label and comment from a workflow_run job #151 pattern. It reads only schema-validated cli_output.v1 / result.v1 and never executes anything from artifacts.

Fork PRs get model runs only after maintainer approval of a pinned SHA (decided).

Budget caps

• Caps come from the config at the base ref. A PR that raises its own cap has no effect until it merges after review.
• A workflow input can only lower a cap: the effective value is min(config, input).
• With CI=true and no cap in config, model steps do not run (CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94: --max-usd is required in CI).
• dokimi implements a–d. e is the documented recommendation for large catalogs.
• Every summary shows: spent / cap for this run, the PR total, and the month total.

Comparison levels: a stored baseline is optional

• L2 is the default. It is always comparable: same model, same day, same harness.
• A stored baseline is a cost optimisation and the source of history. It is not a requirement.
• A new skill runs at L1. A skill with a stale or missing record runs at L2. It does not get inconclusive.

Baselines

• What a baseline is: a named pointer to one run (result.v1 + plan.v1), like a git tag. Its format is the same as any eval result. The difference is its status: a complete, gated run from main.
• Scope: one record per subject (skill/NAME, repo), keyed by skill_sha. skill_sha is the tree hash of the skill directory without its evals. A commit to CI files or docs invalidates nothing.
• No second run after merge. Records are content-addressed. If the merged skill_sha, suite hash, model and grader equal those the PR measured, record promotes the PR result for $0. It reruns only when they differ, or when the PR ran only the smoke plan. Only a run from the approved model job, with the grader from base, can be promoted.
• Update: record advances a baseline only on status: complete and gate: pass, with --expect-previous against races. A failed or inconclusive run on main keeps the old baseline and opens an issue. A manual record dispatch with subject= recomputes a baseline after a grader or suite change.
• Size: about 10–30 KB per record (cells × status × score × tokens × cost, plus hashes). 400 skills come to about 8 MB of current state. Traces are not part of a baseline. compare and gate use cell scores. Traces go to CI artifacts and are only referenced by hash.
• Storage (decided): an orphan branch dokimi-baselines that only record writes. Release assets were rejected: records change many times a week.

  baselines/index.json            # subject → run_id, skill_sha, score, date
  baselines/skill/<name>.json     # current record
  

  One file per subject, so parallel records of different skills never conflict. --expect-previous catches a race on one subject. The branch is squashed periodically. The history chain stays in previous: run_id, and traces stay in CI artifacts. baseline.store: off means live-only (L2).
• Mass invalidation: a new default model or grader marks records stale. revalidate on cron recomputes N subjects per night. Nothing is recomputed in one burst.
• Volume check, intel/skills, 2026-10: 27 commits to main. 9 touched skill content: 5 touched one skill, 3 touched two, 1 touched three. That is about 10–12 record runs a month, not commits × skills.

Repository config: .github/dokimi/

The CLI looks for config in this order: --config, then .github/dokimi/config.yaml, then .dokimi/config.yaml at the root (for repos that prefer it). .agent-eval/ is dropped without a compatibility path. The one external caller, the oneDAL PR, gets updated.

.github/dokimi/
  config.yaml    # dokimi_config.v1: subjects → where their evals are, plan defaults, budgets, baseline store
  policy.yaml    # policy.v1 (#99): what blocks; always read from the base ref
  repo.yaml      # repo_manifest.v1, only for the repository mode (was .agent-eval/repo-eval.yaml)

Evals do not live in .github/. The config points at them:

schema: dokimi_config.v1
defaults:
  skills: "skills/*"              # each match is a subject skill/<name>
  suite: "{skill}/evals"          # convention: evals beside the skill
subjects:
  - id: skill/onetbb
    path: skills/onetbb
    suite:
      harbor: "evaluation/harbor/tasks/onetbb-*"   # central layout, as in intel/skills today
  - id: repo
    kind: repo
    manifest: .github/dokimi/repo.yaml
plan: {models: [sonnet], reps: 3, arms: [none, approved, candidate]}
budget: {pr_max_usd: 5, record_max_usd: 30}
baseline: {store: "branch:dokimi-baselines"}       # or off

• Two layouts are supported: evals beside the skill (no config needed), or a central directory mapped by glob. A task may also declare skills: [NAME] itself.
• skill_sha excludes the evals and any nested skill directory (a nested skill is its own subject; a real dependency is declared in shared:) and suite_hash covers them. A suite may not claim any file of its own skill outside the evals directory; the loader refuses such a config. An edited eval is the separate "new or changed tests" block from User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 and never becomes the new yardstick.
• Impact analysis uses the same file. A changed file maps to a subject or a suite. A file in a shared path triggers the smoke set. A changed file nobody claims gives inconclusive, never a silent skip.
• Config keys use the flag names of cli_flags.py. Locally, dokimi run --config … does what CI does. No secrets in config, ever. .github/dokimi/ should be in CODEOWNERS.
• dokimi_config.v1 must agree with L1 in CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 (plan.v1 + policy.v1): the config holds defaults, and a plan is what a run actually used.

Implementation steps

The workflow contract (inputs, outputs, jobs) is final from the first workflow PR. Steps inside switch to the loop verbs as #94 lands them.

Acceptance

• An external skills repo adds .github/dokimi/config.yaml and one workflow file, and gets a PR verdict with evidence and cost without help (User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 acceptance).
• A planted regression in a skill fails its PR. An inconclusive run never reports pass.
• A fork PR never reaches secrets before a maintainer approves its SHA.
• A merge whose skill was fully measured in its PR costs $0 in record.

Open

• The default arms and reps per budget tier (smoke vs full).
• Whether dokimi assess --init is the only generator of caller files, or dokimi init ci is added under CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 N.

Refs: #121, #94, #99, #97, #138, #151, #156.