Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/campaign-summary-chart.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ two-point default, because it records the whole campaign rather than one
pairing. When neither exists, the report states why no figure was drawn
instead of showing an empty chart.

For runs with `--require-trial-check`, chart generation also requires the
evaluator to publish a comparison in `improvement_pct`. If the checks or the
paired measurements are incomplete, the report labels the medians as raw
measurements and omits both the default and campaign figures. The historical
`summary-chart.json` stays intact. Rebuilding a report removes a previously
generated `summary-chart.svg` when the current evaluation cannot support it.

## Use the command line for a campaign figure

```bash
Expand Down
4 changes: 4 additions & 0 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,3 +263,7 @@ template's copy, validation and catalog-registration steps. Missing hardware
verification remains explicit. Workbench experiments use the
[quickstart](quickstart.md) and [evidence template](../evidence/_template/);
the four CI lanes retain their current roles.

For feedback-coupled models, stochastic audio and streaming/graph experiments,
use the [Omni experiment contract](../src/inferrsi/data/skill_library/workflows/rsi-evidence/references/omni-experiments.md) to select
reference, preflight and executable per-trial qualification checks.
2 changes: 1 addition & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ unvalidated candidates.

- **Inputs:** one JSON specification (chain of accumulating steps, or a set of independent trials); an output path.
- **Scope:** recorded values only. The renderer computes gains, validates the specification and writes a figure; it measures nothing.
- **Output:** an SVG figure, plus a Markdown table with the same values when `--markdown` is given. `report` and `run` also write `summary-chart.svg` into the run directory and inline it in the HTML report.
- **Output:** an SVG figure, plus a Markdown table with the same values when `--markdown` is given. `report` and `run` also write `summary-chart.svg` into the run directory and inline it in the HTML report. When selected trial checks leave the current comparison incomplete, the report omits the figure (including a campaign specification) and removes any previously generated SVG; raw measurements and the campaign JSON remain available.
- **Side effects:** writes the requested files; nothing is executed and no network or GPU is touched.
- **Failure behavior:** exit code 2 with the offending field name when the specification is incomplete, ambiguous or inconsistent; a report without a paired measurement states why it has no figure.
- **Example:** [campaign summary chart](campaign-summary-chart.md)
5 changes: 5 additions & 0 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ plan/run: all doctor flags, plus
--candidate-repo --scenario --mode --agent --executor local
--baseline-command --candidate-command --validation-command
--metric-key --metric-unit --comparison-scope --direction
--require-trial-check (repeatable dotted boolean qualification key)
--repetitions --min-improvement-pct --max-minutes
--workload-id --model --model-revision --output-dir --dry-run
status/report: --run-id --output-dir
Expand Down Expand Up @@ -176,3 +177,7 @@ or `--tier peer`. Export one source explicitly, for example
`--repo-type` names the source owner; vllm_omni queries do not automatically
include its dependencies or peers. See the
[three-tier guide](../src/inferrsi/data/knowledge_packs/README.md) for all sources.

For per-trial completeness, lifecycle and graph qualification, see the
[Omni experiment contract](../src/inferrsi/data/skill_library/workflows/rsi-evidence/references/omni-experiments.md). These optional checks
are frozen in RunSpec and do not replace the quality command.
4 changes: 2 additions & 2 deletions docs/skill-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Generated from the registry, the skill sidecars and the two maintained
matrices. **Do not edit by hand**; regenerate after any registry, skill,
matrix or frozen-plan change.

- Input digest: `sha256:ec4e4b8fbed28d0883fc54ba7bfaf6a76ee0a5f1f28774448f3b49ebed8addb9`
- Input digest: `sha256:f3358ad6ec7526a64d17fa293e937aaa648efa88b7efb485deb98f9f08d07fae`
- Regenerate: `python tools/build_skill_coverage.py`
- Verify without writing: `python tools/build_skill_coverage.py --check`

Expand Down Expand Up @@ -154,7 +154,7 @@ optimization is fully covered or that any procedure has measured benefit.
| [competitor-scan](../src/inferrsi/data/skill_library/workflows/competitor-scan/SKILL.md) | common | Structure a lightweight competitive scan without inventing benchmark numbers or campaign verdicts. | Qualitative positioning of a new RSI repository against other projects before acceptance ... | failure_modes, input_contract, produced_artifact, refusal_point<br>3 advice | N/A: SKILL.md §§ Frame, Rules and Deliverable produce a qualitative competitive comparison with evidence limits, not a hardware-resource decision or implementation mechanism. | — | candidate · not_run · 0 readset(s) | no-Reference-Matrix-row, no-eval |
| [draft-acceptance-issue](../src/inferrsi/data/skill_library/workflows/draft-acceptance-issue/SKILL.md) | common | Draft a campaign acceptance issue with checklist, RunSpec, evidence paths, and placeholders for baseline/candidate SHAs. | Opening a new acceptance campaign in an RSI-style repository, before any baseline or ... | failure_modes, input_contract, produced_artifact, refusal_point<br>3 advice | N/A: SKILL.md §§ Steps and Output prepare an acceptance issue, RunSpec, frozen gates and evidence paths; device identifiers are campaign metadata rather than a hardware-resource mechanism. | — | candidate · not_run · 0 readset(s) | no-Reference-Matrix-row, no-eval |
| [npu-skill-factory](../src/inferrsi/data/skill_library/workflows/npu-skill-factory/SKILL.md) | common | Design and operation of the InferRSI NPU skill factory: artifact lifecycle, skill evaluation contract, corpus ... | InferRSI skill lifecycle management | failure_modes, input_contract, produced_artifact, refusal_point<br>3 advice | N/A: SKILL.md §§ Choose the evidence product, Artifact lifecycle, and What this skill does not do define skill-distillation/evaluation methodology; NPU task tiers are evaluation prerequisites, not a hardware-resource mechanism. | 2 · Methodology and corpus references (Dong1017/OmniRSI @ `codex/skill-eval-bootstrap` (82218b4))<br>2 · Methodology and corpus references (InferMatrixLab/InferRSI PR #9 @ 4ad551a9)<br>2 · Methodology and corpus references (VectorSpaceLab/AREX-Skill @ ac3fe1af)<br>2 · Methodology and corpus references (microsoft/SkillOpt @ 343db22) | candidate · not_run · 14 readset(s) | no-eval |
| [rsi-evidence](../src/inferrsi/data/skill_library/workflows/rsi-evidence/SKILL.md) | common | Guide agents to publish and cite RSI evidence using plan/results/patch/validator paths and PASS/FAIL/INCONCLUSIVE ... | Documenting or reviewing an optimization campaign whose verdict will be cited downstream | failure_modes, input_contract, produced_artifact, refusal_point<br>4 advice | 1 Accelerator · Verify | — | candidate · not_run · 1 readset(s) | no-Reference-Matrix-row, no-eval |
| [rsi-evidence](../src/inferrsi/data/skill_library/workflows/rsi-evidence/SKILL.md) | common | Guide agents to publish and cite RSI evidence using plan/results/patch/validator paths and PASS/FAIL/INCONCLUSIVE ... | Documenting or reviewing an optimization campaign whose verdict will be cited downstream | failure_modes, input_contract, produced_artifact, refusal_point<br>5 advice | 1 Accelerator · Verify | — | candidate · not_run · 1 readset(s) | no-Reference-Matrix-row, no-eval |

## Documents outside the registry

Expand Down
5 changes: 5 additions & 0 deletions src/inferrsi/arguments.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ def parser() -> argparse.ArgumentParser:
default="metrics.latency_ms",
help="Flat or dotted key in benchmark JSON",
)
child.add_argument(
"--require-trial-check", action="append", default=[], metavar="DOTTED_KEY",
help="Require a literal true qualification field in every benchmark JSON; repeatable",
)
child.add_argument("--metric-unit", default="ms")
child.add_argument("--comparison-scope", default="external_command")
child.add_argument("--direction", choices=("minimize", "maximize"), default="minimize")
Expand Down Expand Up @@ -291,6 +295,7 @@ def command(value: str | None) -> tuple[str, ...] | None:
candidate_command=command(args.candidate_command),
validation_command=command(args.validation_command),
metric_key=args.metric_key,
required_trial_checks=tuple(args.require_trial_check),
metric_unit=args.metric_unit,
comparison_scope=args.comparison_scope,
direction=args.direction,
Expand Down
4 changes: 2 additions & 2 deletions src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ These are knowledge abstracts (`--kind knowledge`), with namespaced package
IDs distinct from explicitly selected `checkout_ref` skills. Ascend/CUDA
procedures use their corresponding backend; shared methods use `common`.

Source snapshot: [2c766dbc95605e61b907ce71418ab941287b427a](https://github.com/InferMatrixLab/InferRSI/tree/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library).
Source snapshot: [1e87be46b8d080d439bb516ed9fa9092f8306f62](https://github.com/InferMatrixLab/InferRSI/tree/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library).

```bash
inferrsi knowledge search 'profiling' --repo-type inferrsi --limit 50
Expand All @@ -29,7 +29,7 @@ contain the current input paths and bytes (Git checkout filters are respected).

```bash
python tools/build_skill_index.py --source-revision <committed-source-revision>
python tools/build_skill_index.py --source-revision 2c766dbc95605e61b907ce71418ab941287b427a --check
python tools/build_skill_index.py --source-revision 1e87be46b8d080d439bb516ed9fa9092f8306f62 --check
```

`--check` detects missing, stale and unexpected generated records without writing.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-aclgraph-capture-replay/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-device-health/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-device-health/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-device-health/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-device-health/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-distributed-parallel/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-distributed-parallel/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-distributed-parallel/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-distributed-parallel/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-graph-mode/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-graph-mode/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-graph-mode/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-graph-mode/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-model-adaptation/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-model-adaptation/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-profiling/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-profiling/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-profiling/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-profiling/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-tuning/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-tuning/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-tuning/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-operator-tuning/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-optimization-casebook/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-optimization-casebook/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-optimization-casebook/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-optimization-casebook/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-performance-analysis/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-performance-analysis/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-performance-analysis/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-performance-analysis/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascend-video-diffusion-optimization/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascend-video-diffusion-optimization/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascend-video-diffusion-optimization/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascend-video-diffusion-optimization/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/ascendc-tiling-design/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/ascendc-tiling-design/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/ascendc-tiling-design/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/ascendc-tiling-design/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/hardware/ascend/cann-attribution-overlap/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/hardware/ascend/cann-attribution-overlap/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/hardware/ascend/cann-attribution-overlap/SKILL.md",
"src/inferrsi/data/skill_library/hardware/ascend/cann-attribution-overlap/metadata.json",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
"sources": [
{
"type": "git",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/2c766dbc95605e61b907ce71418ab941287b427a/src/inferrsi/data/skill_library/frameworks/torch-npu/torch-npu/SKILL.md",
"commit": "2c766dbc95605e61b907ce71418ab941287b427a",
"url": "https://github.com/InferMatrixLab/InferRSI/blob/1e87be46b8d080d439bb516ed9fa9092f8306f62/src/inferrsi/data/skill_library/frameworks/torch-npu/torch-npu/SKILL.md",
"commit": "1e87be46b8d080d439bb516ed9fa9092f8306f62",
"paths": [
"src/inferrsi/data/skill_library/frameworks/torch-npu/torch-npu/SKILL.md",
"src/inferrsi/data/skill_library/frameworks/torch-npu/torch-npu/metadata.json",
Expand Down
Loading
Loading