Repository navigation
[Roadmap] InferRSI: Optimization, Evaluation, and Engineering #316
Description
Goal
Track nine work items across six dimensions, covering InferRSI's optimization capabilities, knowledge accumulation, evaluation, and engineering delivery. The table preserves the original item numbers for reference.
Related overall roadmap: #61.
Workstreams and Ownership
| Dimension | Original item | Work item and focus | Owner | Tracking issue |
|---|---|---|---|---|
| 1. Performance Ceilings and Search Capabilities | 1 | Performance ceiling analysis and kernel performance optimization: Analyze task-level performance limits and the gap from the current implementation to guide bottleneck diagnosis and kernel optimization. | @fake0fan | #303 |
| 1. Performance Ceilings and Search Capabilities | 5 | Engine-side search space optimization: Improve search spaces and optimization goals, extending coverage to torch / torch_npu. | @amy-why-3459 | #143 |
| 2. Knowledge Retrieval and Evaluation | 2 | Knowledge base indexing and evaluation: Evaluate both indexing / retrieval performance and accuracy, including whether retrieved knowledge is correct and useful for optimization tasks. | @Dong1017 | #273 (association pending verification; see notes below) |
| 3. Benchmarks and Effectiveness Evaluation | 4 | Benchmarks (#248 + HarnessPro): Build reproducible tasks and controlled evaluations to measure InferRSI's impact on optimization outcomes, process reliability, and cost. | @Bounty-hunter | #248; HarnessPro |
| 3. Benchmarks and Effectiveness Evaluation | 9 | Test Experience Collection: Collect test cases, issues encountered, and resolution experience, and consolidate them into reusable testing knowledge. | @yenuo26 | Tracked in this issue |
| 4. Observability and Human Feedback | 3 | Dashboard: Observe the full RSI optimization workflow, capture and correlate traces, and inspect stages, iteration results, and problems. | @david6666666 | #314 (sub-issue of this roadmap) |
| 4. Observability and Human Feedback | 7 | Manual trace review and framework feedback: Everyone reviews optimization traces, diagnoses where underperforming tasks go wrong, and feeds findings back into the framework, knowledge base, and search strategies. | All contributors | Tracked in this issue, using the trace capabilities in #314 |
| 5. Usability and Release Engineering | 6 | Agent Plugins + Release Bot: Improve integration and usability in Codex, Claude Code (CC), and openJiuwen, alongside automated releases and delivery. | @zengchuang-hw | Agent plugins: #152; release bot: #238 |
| 6. New Model Integration and Reusable Experience | 8 | 0 Day Model Support: Expand integration, validation, and optimization experience across Transformers → GPU Inference Engine → NPU Inference Engine. | @amy-why-3459 | #315 (sub-issue of this roadmap) |
How the Workstreams Connect
- Knowledge and search support optimization: Knowledge retrieval, new model integration experience, and engine search spaces provide actionable optimization candidates.
- Evaluation and observability support decisions: Benchmarks validate optimization outcomes, while the dashboard and traces explain the process and where problems occur.
- Human review drives improvement: Contributors use traces to diagnose failures or limited gains, turn feedback into framework changes, knowledge additions, or search strategy updates, and evaluate the results.
Tracking Notes
- Item 2: Keep [Investigation] Agent-led workflow gaps: closure, decision visibility, review provenance and focus #273 as requested, with its association pending verification. Its current topic is an investigation into agent collaboration workflows, which does not fully match knowledge base indexing and evaluation. The tracking association needs to be aligned later.
- Item 6: This roadmap records Codex / Claude Code / openJiuwen as the target hosts. The current body of [RFC] Agent plugins: a consistent InferRSI workflow in Codex, DSH and openJiuwen #152 lists Codex / DSH / openJiuwen; the host scope needs to be aligned in that issue.
- [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 currently focuses on performance ceiling modeling and gap metrics. Kernel performance optimization is the implementation goal for this roadmap's corresponding workstream, with specific tasks to be refined in the relevant issues.
- Existing issues remain the source of implementation progress and acceptance status; this roadmap summarizes workstreams and owners. Items 3 and 8 have dedicated sub-issues, while item 7 is an ongoing feedback activity for all contributors.
Activity
[-][Roadmap] InferRSI：优化能力、评测闭环与工程化[/-][+][Roadmap] InferRSI: Optimization, Evaluation, and Engineering[/+]agent-ci-infra-bot commentedon Nov 2, 2026
该 Issue 用六个维度/八项工作汇总本轮方向、负责人与跟踪 issue，属于协调型 Roadmap，全文只做「汇总方向与负责人」，没有要求任何代码改动。我已核对 #61/#303/#143/#273/#248/#314/#315/#152/#238 都存在且标题与表格大体对应，但多条跟踪入口与所述范围、当前状态并不一致。Verdict：不需要代码改动（no code change requested）；在把本 Issue 当作跟踪入口之前，工作项 2、4、1/5、7 的跟踪与验收位置，以及它与 #61 的从属关系需要先补齐（详见 findings）。Labels: kind:roadmap, area:process（程序级 roadmap 锚点，内容为仓库级协作与治理，不是某项代码交付）
工作项 2 的跟踪条目与所述范围不一致
status:proposed），与「索引/检索的性能与精准度」没有对应内容。Issue 正文已标注「关联待核对」，但表格仍把它当作工作项 2 的跟踪入口。工作项 4 的跟踪条目当前处于中止状态，表格未反映
docs/harness-improvement.md:9-16只把 HarnessPro 列为离线 harness 改进 MVP 的设计来源（trace 驱动诊断与质量/成本选择），并不是 benchmark 交付物。工作项 1、5 指向 RFC，缺少落地任务
kind:rfc，正文自述「No implementation, no measurement and no evidence bundle exists for it」；[RFC] Hardware-aware optimization goals throughout the RSI workflow #143 同为kind:rfc，无 assignee、Decision needed: none、无新评论，主题是「硬件感知的优化目标」，而不是「完善搜索空间并把 torch / torch_npu 纳入覆盖」。Issue 正文说「具体任务继续在相关 issue 中细化」，但没有点名任何落地 issue；src/inferrsi/catalog.py:10的REPO_TYPES仍是 7 个引擎类型，torch/torch_npu只作为 knowledge scope 出现（src/inferrsi/catalog.py:17）。docs/agent-collaboration.md「assignment identifies accountability」不一致。工作项 6 的宿主范围与 #152 冲突且无决策记录
status:proposed。工作项 7 没有跟踪入口、单一 owner 或验收判据
docs/agent-collaboration.md的角色定义无法判定完成或阻塞，也不会产生可验收的产出；「全体成员」等同无人负责。kind:trackingissue 并指定负责人。与 #61 的关系、阶段与子 issue 关系未定义
kind:roadmap，自带 Now/Next/Later、Issue organization 与 related-work 表（其中已含 task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248、[RFC] Benchmark selection by task for InferRSI #60、[RFC] Skill Eval v1: revision-scoped discovery and task-efficacy evaluation #40 等），与本 Issue 的八项存在重叠，但本 Issue 未说明两者的权威/从属关系；六个维度也没有阶段或依赖顺序（kind:roadmap的描述是「Program-level roadmap anchor (Now / Next / Later)」）。另外 [Task] RSI Dashboard: Optimization Workflow Visualization and Trace Capture #314/[Task] 0 Day Model Support: Reusable Integration Experience from Transformers to GPU and NPU Inference Engines #315 只在正文写「Parent issue: [Roadmap] InferRSI: Optimization, Evaluation, and Engineering #316」，GitHub 的sub_issues_summary.total对 [Roadmap] InferRSI: Optimization, Evaluation, and Engineering #316 仍为 0，子关系只存在于文字里。Reviewed commit: 2baebf6
Coverage: Snapshot: 2637 files, 42149780 bytes; skipped 0 entries; Initial model context: 28 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history