Skip to content

Qualify repeated tool advice with input and safety evidence - #335

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/repeated-tool-evidence
Nov 18, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/repeated-tool-evidence

Conversation

@dipeshbabu

@dipeshbabu dipeshbabu commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

In the documented retrieval pilot, a repeated tool name led to a concurrency recommendation. The calls read the same immutable section. Request-local reuse removed the extra work with no loss of correctness. The plan needs to explain what the trace establishes before proposing a change.

Changes

  • Name-only evidence identifies the repetition and asks the developer to compare inputs, dependencies and shared state. Patch plans offer the same investigation steps.
  • A successful serial group with matching opaque input keys and declared read-only behavior suggests testing request-local reuse. The estimate uses the recorded first call and carries its own formula and assumptions.
  • Explicit concurrency declarations still support a concurrency trial. Existing dependency and safety exclusions apply before selecting a recommendation.
  • Name-only findings retain their saved IDs. The reuse proposal has a distinct identity. Trace schema, finding family and legacy graph projections remain compatible.

Validation

  • Added evidence and patch-plan regressions for declared reuse, mixed keys, malformed declarations and failed calls. Updated the affected HTML snapshot.
  • .venv/Scripts/python.exe -m pytest tests/test_parallelization_evidence.py tests/test_patch_plan.py tests/test_estimates.py tests/test_finding_benchmarks.py tests/test_finding_rules.py tests/test_html_report.py -q --basetemp D:/agentloop/runs/repeated-tool-focused-final --tb=short — 102 passed. This includes the preserved inferred/declared finding identity and the changed HTML report language.
  • .venv/Scripts/python.exe runs/bugfix_quality_checks.py — all applicable hooks passed.
  • uv build --out-dir runs/repeated-tool-dist --quiet — wheel and source distribution built.
  • All runnable hosted checks passed on final head 562d1092e2735fec3af68e158f062928de008c35, including the four full Python suites, native wheel smoke, PostgreSQL, dependency review and CodeQL.
  • .venv/Scripts/python.exe -m pytest -q --basetemp D:/agentloop/runs/repeated-tool-full --tb=short — 2,815 passed, 109 skipped.

Scope and limitations

The first rule applies to groups of three or more calls in one parent scope. Reuse requires the developer to verify that the key covers all effective inputs, the data is stable and independent task checks pass. The report's predicted time savings are conditional. Developers apply the change and verify it with replay.

Substantial implementation and validation assistance was provided by Codex.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_2ecc799676464bda runs/research_agent_baseline.json yes agentloop_demo
candidate run_34701e9917db4866 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_7de004e3a190: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu marked this pull request as ready for review November 18, 2026 05:25
@dipeshbabu
dipeshbabu merged commit f90f0d2 into main Nov 18, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/repeated-tool-evidence branch November 18, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant