Skip to content

Show trace latency and runner wall time in guided reports - #334

Merged
dipeshbabu merged 5 commits into
mainfrom
codex/guided-runner-timing
Nov 18, 2026
Merged

dipeshbabu merged 5 commits into
mainfrom
codex/guided-runner-timing

Conversation

@dipeshbabu

@dipeshbabu dipeshbabu commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

The guided retrieval pilot measured a large improvement inside the trace and a much smaller improvement across the runner process. The report exposed only the trace duration, which made the two scopes easy to confuse.

Changes

  • Time each runner from launch attempt through exit with a monotonic clock. The boundary includes startup, trace export and interruption cleanup. The capture receipt keeps its original timestamps.
  • Show trace latency and runner wall time in the CLI and the JSON, Markdown and HTML reports. Paired medians use compatible measurements from completed, correct pairs. Per-attempt details retain available timings for rejected and incomplete attempts.
  • Label the actual comparison as a control when capture uses interleaving. Keep runner timing descriptive and preserve the trace-based latency gate and quality checks.
  • Read older capture timestamps as a separate, labeled timing source. Invalid or missing measurements stay unavailable. Existing trace, capture and report schema versions and fields retain their meanings.

Validation

  • Added regression coverage for interleaved source selection, legacy timestamps, invalid timing, missing traces, failed quality and the case where a trace improves while elapsed runner time regresses.
  • .venv/Scripts/python.exe -m pytest tests/test_improvement.py tests/test_improvement_init.py tests/test_guided_validation.py -q --basetemp D:/agentloop/runs/guided-runner-timing-focused --tb=short — 56 passed, 1 skipped. The Windows smoke and onboarding test initially decoded redirected UTF-8 with the system code page. Both subprocess readers now explicitly use UTF-8.
  • .venv/Scripts/python.exe -m pytest tests/test_improvement.py::test_timing_summary_reads_legacy_receipts_and_keeps_runner_time_out_of_gates tests/test_improvement.py::test_cli_runs_the_normal_entry_points_and_returns_quality_failure tests/test_improvement_init.py::test_complete_offline_onboarding_with_manual_edit_and_rejected_candidate -q --basetemp D:/agentloop/runs/guided-timing-final-focused -W error::pytest.PytestUnhandledThreadExceptionWarning — 3 passed after the UTF-8 fix and report wording changes.
  • .venv/Scripts/python.exe runs/bugfix_quality_checks.py — all applicable pre-commit checks passed. A concurrent wheel build briefly created a temporary source tree during one link check. Running the check after the build resolved it.
  • uv build --out-dir runs/guided-timing-final-dist --quiet and uv tool run --from twine twine check runs/guided-timing-final-dist/agentloop_profiler-0.2.3-py3-none-any.whl runs/guided-timing-final-dist/agentloop_profiler-0.2.3.tar.gz — both artifacts passed.
  • .venv/Scripts/python.exe -m mkdocs build --strict --site-dir D:/agentloop/runs/guided-timing-site — passed.
  • The locally installed wheel recaptured 16 held-out retrieval pairs and 4 deliberately wrong pairs. All 16 correct pairs passed. All 4 wrong pairs were rejected and excluded from the timing summary, with per-attempt measurements retained. Recomputing a retained 0.2.3 comparison preserved its result and reported its timestamp-derived runner durations. Chrome verified both final reports at 390px and 1440px without document overflow.
  • runs/guided-timing-validation/venv/Scripts/python.exe -I scripts/smoke_installed_wheel.py --expected-version 0.2.3 passed from a fresh environment with the locally built wheel.
  • Pilot and report verification ran venv/Scripts/python.exe -I replay_pilot.py and venv/Scripts/python.exe -I check_final_reports.py from D:/agentloop/runs/guided-timing-validation. Browser rendering was checked with uv run --isolated --no-project --with playwright==1.65.0 python runs/guided-timing-validation/check_html.py.
  • The held-out capture showed a median paired trace difference of −52.911 ms and a runner difference of +23.855 ms while other local checks shared the host. The CLI and report make both scopes visible.
  • .venv/Scripts/python.exe -m pytest -q --basetemp D:/agentloop/runs/guided-timing-full --durations=5 — 2,808 passed and 109 skipped. This run collected tests before the UTF-8 test-reader fix, so it retained two reader warnings. The final focused rerun treated those warnings as errors and passed.
  • Final commit 4af75b0bcc2bc2a42ad10bc81592f14e67cfc02a passed all 23 runnable hosted checks. GitHub Pages deployment runs on main. The automatic CodeRabbit review was skipped under its existing eligibility policy and supplied no approval.

Limits

The offline retrieval fixture measures an owned filesystem workload on one host. Its speedup describes that workload. External developer onboarding remains unmeasured. Runner wall time covers the child process boundary and excludes capture-side parsing, validation and report generation.

Substantial implementation and validation assistance was provided by Codex.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.68%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.68% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_3634a6658e994f0b runs/research_agent_baseline.json yes agentloop_demo
candidate run_68a98985d1954400 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.68%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_4d4f2c4d19b1: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu marked this pull request as ready for review November 18, 2026 03:44
@dipeshbabu
dipeshbabu merged commit 373c4c8 into main Nov 18, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/guided-runner-timing branch November 18, 2026 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant