[surajshetty3416]

Summary

Builder AI sends no usage telemetry today. The only event, builder_ai_used, was captured in ai_page_generator.py, and the move to builder/ai/ dropped it. This PR captures it again: one event per agent turn, with what happened in that turn.

 run_agent_job
   runner = AgentRunner(...)
-  runner.run()
+  try:
+    runner.run()          # each exit sets runner.outcome
+  finally:
+    runner.capture_usage()  # capture("builder_ai_used", "builder", user=..., properties=...)

outcome is set where each turn already ends:

finish_turn      -> complete | incomplete (hit max rounds) | empty
terminal card    -> awaiting_input
emit_cancelled   -> cancelled
fail_turn        -> error
session locked   -> busy
raised out       -> failed

Properties: outcome, stop_reason, provider, model, effort, tools (sorted tool names), operations, generated_page, rounds, llm_calls, prompt_tokens / cached_tokens / completion_tokens, tool_failures, with_image, elapsed_ms, page, session.

• Custom gateways stay anonymous. ModelRegistry.telemetry_route reports the preset prefix (frappe, openrouter, anthropic, …) or custom. A custom gateway's prefix is a name the site chose, so it is never sent.
• Evals send nothing. The capture lives in run_agent_job, so the eval harness, which drives AgentRunner directly, doesn't emit events.
• A telemetry failure never fails a turn. It is logged at warning level.
• Resumed turns count too. A turn resumed after a confirm card goes through run_agent_job, so it is captured.

Evidence

• Before: git grep builder_ai_used on develop finds nothing. In Pulse, the event's last occurrence was 2026-08-25 (62 events from 6 sites, all with empty properties). So far there's no way to chart AI adoption, model mix or failure rate.
  After: new tests in builder/ai/test_agent_jobs.py:

test_a_finished_turn_reports_its_ai_usage      one builder_ai_used, outcome=complete, tools, generated_page, provider/model
test_a_turn_that_crashes_is_still_reported     run_turn raises -> still captured, outcome=failed
test_usage_never_names_a_custom_gateway        gateway/llama-4 -> ("custom", "llama-4")
test_broken_telemetry_does_not_fail_the_turn   capture raises -> capture_usage swallows it

builder.ai.test_agent_jobs: Ran 8 tests ... OK

All other builder/ai/test_* modules pass locally. The exception is test_preview_image: its setUp patches frappe.utils.chromium, which my local Frappe checkout predates, so it errors there with or without this change.

Merge Danger

Door: two-way

This only adds an event. Reverting removes it, and nothing reads it yet apart from the Insights dashboard that will chart it.

Blast Radius: telemetry

Only sites with Pulse enabled send it (pulse_api_key set, telemetry on, no developer mode). One event per turn, queued by the Pulse client and not sent inline, so it adds no latency to the turn.

🤖 Generated with Claude Code

[mergify]

This pull request is closed, so it cannot be queued. Reopen it to get the box back.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.42%. Comparing base (86b6975) to head (96da7ca).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1084   +/-   ##
========================================
  Coverage    81.42%   81.42%           
========================================
  Files           70       70           
  Lines         7541     7541           
========================================
  Hits          6140     6140           
  Misses        1401     1401           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.