Ensemble — CSV Data-Analysis Agent

A reason → plan → act → observe → respond agent that answers questions about a CSV file you upload, streamed live in a Gradio chat UI.

Why this domain

A CSV data-analysis agent is fully self-contained: it needs no external API (unlike a PubMed research agent) and no fabricated log data (unlike an SRE agent) — the tool call (Python code execution against the uploaded file) is native to the domain itself. This keeps the graded surface area small while still exercising every required capability: tool routing between two distinct tools, structured decision-making, ambiguity handling (asking for clarification instead of guessing), and self-critique over a generated answer.

Architecture

flowchart TD
    U[User: CSV upload + query] --> UI[Gradio ChatInterface - app.py]
    UI --> AGENT[agent.run_agent loop]
    AGENT -->|thought / decision| LLM[Anthropic Claude via llm.py]
    LLM -->|action: tool_call| ROUTE{which tool?}
    ROUTE -->|get_schema| SCHEMA[tools.get_schema]
    ROUTE -->|run_python| GUARD[guardrails.check_code_safety]
    GUARD -->|safe| EXEC[tools.run_python: sandboxed exec]
    GUARD -->|unsafe| REJECT[ToolResult ok=False]
    SCHEMA --> OBS[observation StepEvent]
    EXEC --> OBS
    REJECT --> OBS
    OBS --> AGENT
    LLM -->|action: final_answer| CRITIQUE[llm.get_critique self-critique]
    CRITIQUE -->|approved| FINAL[final StepEvent]
    CRITIQUE -->|rejected| AGENT
    LLM -->|action: clarify| CLARIFYQ[clarify StepEvent]
    FINAL --> UI
    CLARIFYQ --> UI
    AGENT -. every StepEvent .-> TRACE[TraceWriter -> runs/*.jsonl]

Loading

• src/agent.py — the loop itself (run_agent), transport-agnostic: an async generator of StepEvents, consumed by both app.py and tests/evaluate.py. Bounded to settings.max_iterations (default 5).
• src/llm.py — the only module that talks to Anthropic. Gets structured output (AgentDecision, CritiqueResult) via forced tool-choice, with retry/backoff and a timeout.
• src/tools.py / src/guardrails.py — the two tools (get_schema, run_python) and the AST-based allowlist guard-rail + CSV validation that runs before any generated code executes.
• src/models.py — every structured type (tool calls, tool results, the per-step trace event, the final answer) as Pydantic models.
• prompts/ — system prompt and few-shot examples, kept out of code.
• app.py — the only entrypoint. A gr.ChatInterface (multimodal: CSV upload + text in one turn) whose callback is a streaming generator — Gradio's own recommended pattern for incremental output, no SSE/WebSocket layer needed.

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# then edit .env and set your real ANTHROPIC_API_KEY

Run the app

python app.py

Opens at http://127.0.0.1:7860. Upload a CSV, ask a question; follow-up questions in the same session reuse the uploaded file. Every turn's reasoning/tool-call/observation/final steps stream into the chat live and are written to runs/<timestamp>.jsonl.

Run the eval

python tests/evaluate.py

Drives src/agent.py directly against the 4 scenarios in tests/scenarios.json (simple aggregate, ambiguous query, bad input, multi-step analysis) and prints a PASS/FAIL/SKIPPED table. Three of the four scenarios run against the checked-in fixture at tests/data/orders.csv (columns order_id, product, category, quantity, price, order_date, matching prompts/few_shot.md's worked examples, ~4,300 rows spanning 2023–2024 so the multi-step month-over-month scenario has real trend structure to find). Swap in your own CSV there and they'll still run against it; if the file is ever missing, those scenarios report SKIPPED rather than failing. The bad_input scenario needs no file — it deliberately targets a nonexistent path to exercise the guard-rail.

Project structure

app.py                # Gradio ChatInterface entrypoint
src/
  agent.py            # reason->plan->act->observe->respond loop + TraceWriter
  tools.py             # run_python, get_schema
  guardrails.py        # AST allowlist check_code_safety, validate_csv
  llm.py               # Anthropic client: structured output, retries, timeout
  models.py            # Pydantic schemas
  config.py            # env-driven settings
prompts/
  system.md
  few_shot.md
tests/
  scenarios.json
  evaluate.py
runs/                  # saved traces (JSONL per run)
requirements.txt
.env.example