From 56ea6575a2ac22c7f4c8b2f6ad35a702ca969dbf Mon Sep 17 00:00:00 2001 From: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> Date: Fri, 9 Oct 2026 23:13:30 +0800 Subject: [PATCH 1/2] docs(skills): route benchmark effectiveness and efficiency analysis Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> --- skills/loopx-benchmark/SKILL.md | 46 +++++++++++++++++-- skills/loopx-benchmark/agents/openai.yaml | 4 +- skills/loopx-performance-diagnosis/SKILL.md | 10 +++- .../agents/openai.yaml | 2 +- 4 files changed, 54 insertions(+), 8 deletions(-) diff --git a/skills/loopx-benchmark/SKILL.md b/skills/loopx-benchmark/SKILL.md index 0f3a4b3e4c..8e0bdcc545 100644 --- a/skills/loopx-benchmark/SKILL.md +++ b/skills/loopx-benchmark/SKILL.md @@ -1,6 +1,6 @@ --- name: loopx-benchmark -description: "Use when acting as the operator or post-run analyst of a LoopX-managed benchmark experiment through benchmark-toolkit: select or launch runs, maintain experiment-board rows, qualify integrity, or analyze matched comparisons and case insights. Solving an assigned benchmark task under an existing runner does not by itself select this skill. Excludes casual benchmark discussion and ordinary software microbenchmarks." +description: "Operate or analyze LoopX-managed benchmark experiments through benchmark-toolkit, including trajectory-based effectiveness and efficiency diagnosis, matched comparisons, run selection and integrity. Solving an assigned task under an existing runner does not by itself select this skill. Excludes casual benchmark discussion and ordinary software microbenchmarks." --- # LoopX Benchmark Workflow @@ -81,12 +81,49 @@ explicitly exploratory and retain the relevant limitations and counterexamples. - **Monitor an active campaign:** read the board and runtime-owned projections; update only on material run transitions. Do not manufacture progress from a timer tick. +- **Diagnose effectiveness or trajectory efficiency:** follow the analysis below. + For active runs, use authorized solver/runtime observations and released score + projections; keep findings provisional and hidden evaluator evidence closed. - **Analyze a terminal run:** wait until solving is terminal and scoring is complete before reading hidden evaluator evidence or writing a case insight. For a generic library microbenchmark or an eval with no LoopX Goal/board, use the task's normal tools instead of imposing this workflow. +## Diagnose effectiveness and trajectory efficiency + +Start with the task's real success criterion and evaluator behavior, then explain +which work produced useful progress. Keep score quality and operational efficiency +separate: fewer bytes, calls or tokens are not proof of a better task outcome. + +1. Read the board and align source, model, task, budget, feedback mode, sampling, + evaluator and concurrency. Compare common elapsed windows and label unmatched + history as diagnostic. Missing or invalid scores remain distinct from zero; + use the task's native ranking when identifying a retained best result. +2. Inspect representative early, middle and late trajectory segments, including + stalls and counterexamples. Record the selection rule and sample denominator. + Connect actions to changed artifacts, validation and scored snapshots; count + planning, control reads, tool recovery and actual task work separately. Use + hidden task/evaluator evidence only after solver and scoring are terminal. +3. Separate candidate causes: task difficulty or strategy, model token throughput, + control interaction overhead, tool failures, and evaluator/feedback delay. + Align artifact capture, grading and delivery times with solver activity. + Compare both wall time and useful progress per token/active work interval where + measured; a score slope or a token-rate difference alone cannot identify the + cause. Preserve unknowns when telemetry is absent. +4. Turn the strongest supported cause into a small discriminating intervention + or ablation within existing authority. Reuse current qualified baselines and + change one relevant factor where possible; disclose unavoidable confounds. + Prioritize expected task benefit and technical depth over packet size or PR + count. If repeated reads are wasteful, verify the consumer's obligations before + reducing them; a planning or recovery change must still lead to useful work. +5. Report observations, causal hypotheses and validated effects distinctly, + including regressions and remaining uncertainty. Use + [loopx-performance-diagnosis](../loopx-performance-diagnosis/SKILL.md) only for + an evidenced owned-process cost; return its measurements to this task-level + analysis. Do not substitute profiler hotspots for outcome evidence or require + profiling to inspect duplicate content and unnecessary interactions. + ## Experiment sequence 1. **Read the experiment board before launching or selecting a case.** @@ -181,9 +218,10 @@ the task's normal tools instead of imposing this workflow. confirmed disclosure plus causal use disqualifies the score. - `classify-artifacts` classifies benchmark artifact paths without reading them; use it before reading or publishing any candidate artifact. -- The solver lane must not read hidden tests, verifier sources, gold answers, or - official feedback during the solving phase. The post-run analyst may read full - private evidence only after the solver is terminal and scoring is complete. +- The solver lane must not read hidden tests, verifier sources or gold answers. + During solving, official feedback is limited to what the declared run protocol + releases to that solver. The post-run analyst may read full private evidence + only after the solver is terminal and scoring is complete. - `capability bind` selects an external provider implementation for a Goal; it is not the activation mechanism for this builtin capability. Todo `required_capability` fields remain runtime prerequisites, not product diff --git a/skills/loopx-benchmark/agents/openai.yaml b/skills/loopx-benchmark/agents/openai.yaml index 913b4b6327..55deaf2d0c 100644 --- a/skills/loopx-benchmark/agents/openai.yaml +++ b/skills/loopx-benchmark/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "LoopX Benchmark" - short_description: "Operate LoopX-managed benchmark experiments" - default_prompt: "Use benchmark-toolkit for this LoopX-managed experiment: choose the correct operator lane, read the experiment board before run selection, preserve solver/analyst boundaries, preview writes, and compare only countable matched arms." + short_description: "Analyze benchmark outcomes, trajectories and efficiency" + default_prompt: "Use $loopx-benchmark to operate or analyze this managed experiment: compare qualified outcomes and trajectory efficiency, preserve solver/analyst boundaries, and distinguish causal evidence from runtime profiling." diff --git a/skills/loopx-performance-diagnosis/SKILL.md b/skills/loopx-performance-diagnosis/SKILL.md index 5aac749902..3ff8421c14 100644 --- a/skills/loopx-performance-diagnosis/SKILL.md +++ b/skills/loopx-performance-diagnosis/SKILL.md @@ -1,10 +1,18 @@ --- name: loopx-performance-diagnosis -description: Diagnose an owned slow command or runtime with language-appropriate profilers, independent baseline measurements and local-private evidence. Use for demonstrated CPU, wall-time, allocation or IO regressions; profiling does not qualify a budget or grant execution authority. +description: Diagnose demonstrated CPU, command latency, allocation or IO regressions in an owned process with controlled measurements and language-appropriate profilers. Route benchmark effectiveness, score growth and trajectory efficiency to loopx-benchmark; profiling does not qualify a budget or grant execution authority. --- # LoopX performance diagnosis +First identify the question. For a LoopX-managed benchmark's outcome quality, +score growth, useful progress per time/token, or agent trajectory, use +[loopx-benchmark](../loopx-benchmark/SKILL.md). A flatter score curve, repeated +reads or more tokens alone do not establish a process-performance regression. +Keep that analysis with the benchmark owner; use this skill as a focused subtask +only when evidence identifies a slow owned command or resource cost. Do not +require a profiler before analyzing effectiveness or interaction overhead. + Read the current Goal/Todo contract and repository optimization evidence rules. State the real user operation, observed failure and owning acceptance. Preserve the exact source, runtime/interpreter, backend/history, inputs and concurrency. diff --git a/skills/loopx-performance-diagnosis/agents/openai.yaml b/skills/loopx-performance-diagnosis/agents/openai.yaml index bca67aa527..b07d5a951e 100644 --- a/skills/loopx-performance-diagnosis/agents/openai.yaml +++ b/skills/loopx-performance-diagnosis/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "LoopX Performance Diagnosis" short_description: "Profile slow commands with controlled local evidence" - default_prompt: "Use $loopx-performance-diagnosis to profile an owned slow command, compare an independent baseline, and keep raw profiles local." + default_prompt: "Use $loopx-performance-diagnosis for an observed process cost with controlled measurements; route benchmark outcomes and trajectory efficiency to $loopx-benchmark." From 1425603fee4d8aba5d2383975ab07fdbd5aa6ad8 Mon Sep 17 00:00:00 2001 From: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> Date: Fri, 9 Oct 2026 23:43:20 +0800 Subject: [PATCH 2/2] docs(skills): include active-run analysis in benchmark introduction Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> --- skills/loopx-benchmark/SKILL.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/skills/loopx-benchmark/SKILL.md b/skills/loopx-benchmark/SKILL.md index 8e0bdcc545..4dbe980697 100644 --- a/skills/loopx-benchmark/SKILL.md +++ b/skills/loopx-benchmark/SKILL.md @@ -12,9 +12,10 @@ integrity boundaries. This packaged skill is its task-triggered Agent playbook. An assigned solver follows its task instructions and current execution contract. The words “benchmark”, “evaluation”, or “submission” in that task do not grant the operator role or require experiment-board discovery. Use this workflow when -the requested work actually includes run management or post-run analysis; an -explicit request to use the skill still applies. Keep the solver's task-local -validation and authorized submission path distinct from experiment management. +the requested work actually includes run management, analysis of an active run, +or post-run analysis; an explicit request to use the skill still applies. Keep +the solver's task-local validation and authorized submission path distinct from +experiment management. The capability is catalog-ready without a per-Goal enable switch. Installing this skill does not grant runner, shell, network, credential, private-evidence,