Context
webcmd needs a repeatable evaluation methodology for browser automation and command-generation workflows. We should be able to compare webcmd against adjacent projects/libraries on the same sample tasks and get actionable feedback about where webcmd performs better or worse.
Primary competitors / comparison targets:
Goal
Create an eval framework that runs representative browser automation tasks across webcmd and competitor tools, records each agent/library's path through the task, and reports deterministic results only. The harness should not use an LLM as a judge. Results should come from reproducible task assertions, fixture/live-site checks, and numeric telemetry such as tokens used, time taken, action counts, retries, and failures.
Proposed scope
- Define a small benchmark suite of representative browser automation tasks.
- Run each task through webcmd and comparison tools under consistent conditions.
- Support a comparison matrix across agents and model tiers with different intelligence/cost/latency profiles.
- Capture enough trace data to inspect how each system solved the task, including browser actions, page observations, network/context usage where available, final output, failures, retries, token usage, cost where available, and timing.
- Score task success through deterministic checks, such as exact/structured output assertions, expected page state, downloaded artifact checksums/metadata, API response validation, or fixture state validation.
- Store eval results in a machine-readable format that can be compared over time and used in CI or scheduled regression runs.
- Generate a human-readable metrics summary without relying on subjective judge feedback.
Open questions
- Which first-party tasks best represent webcmd's intended use cases?
- Should evals run against live websites, local fixture sites, or both?
- How do we normalize traces and telemetry across tools with different observability models?
- Which agent/model matrix should we start with, and how should we group models by intelligence/cost/latency tier?
- Which deterministic thresholds should block releases versus only produce advisory feedback?
Acceptance criteria
- A documented deterministic eval methodology exists in the repo.
- At least one runnable eval suite compares webcmd against at least one competitor.
- The suite can run a matrix of agent/model configurations with different intelligence/cost/latency profiles.
- Eval outputs include machine-readable metrics for pass/fail, tokens used, wall-clock time, retries, failures, and other relevant deterministic telemetry.
- Human-readable summaries are derived from deterministic metrics, not an LLM judge.
- The framework is designed so new tasks, competitors, and model/agent configurations can be added incrementally.
Context
webcmd needs a repeatable evaluation methodology for browser automation and command-generation workflows. We should be able to compare webcmd against adjacent projects/libraries on the same sample tasks and get actionable feedback about where webcmd performs better or worse.
Primary competitors / comparison targets:
Goal
Create an eval framework that runs representative browser automation tasks across webcmd and competitor tools, records each agent/library's path through the task, and reports deterministic results only. The harness should not use an LLM as a judge. Results should come from reproducible task assertions, fixture/live-site checks, and numeric telemetry such as tokens used, time taken, action counts, retries, and failures.
Proposed scope
Open questions
Acceptance criteria