Portable results: an EvalPort adapter for src/aggregators output?

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange spec for eval data (TestCase/Grader/Result/ResultSet/GraderResult, Python + TS SDKs). Posting here rather than opening an issue since this is a "would you want it" question, not a bug.

I went through src/aggregators/aggregate_results.py and the tasks/<mcp>/<suite>/<category>/<task_id>/ layout, and the mapping onto EvalPort's model is close to 1:1:

• tasks/<mcp>/<suite>/<category>/<task_id>/{description.md,verify.py} → TestCase.input (description) + a code-type Grader (verify.py is a sandboxed grading function, which is exactly what EvalPort's code grader type models — no lossy translation needed).
• results/<exp>/<model>__<mcp>/run-N/<category>__<task>/meta.json's execution_result.{success,error_message} → GraderResult{passed, score, metadata.error_message}.
• The rest of meta.json (agent_execution_time, token_usage.{input,output,total}_tokens, turn_count, actual_model_name) → Result.metadata, unchanged.
• The run-1..k structure that aggregate_results.py already reduces to pass@1/pass@k/pass^k → a ResultSet per model__mcp, with metadata.num_repetitions = k.

That last one isn't hand-wavy on my end — how ResultSet should represent repeated-attempt evals (num_repetitions/epochs) is an actually-open question in EvalPort's own spec right now (discussion #22), and MCPMark's pass@k/pass^k over k runs is real production data for that question, not a hypothetical.

Sketch of the converter (meta.json → EvalPort dicts):

def task_to_testcase(service: str, category: str, task_id: str, description_md: str) -> dict:
    return {
        "id": f"{service}__{category}__{task_id}",
        "input": description_md,
        "graders": ["gr_verify"],
        "metadata": {"mcp_service": service, "category": category},
    }

def meta_to_result(task_case_id: str, meta: dict) -> dict:
    exec_result = meta.get("execution_result", {})
    success = bool(exec_result.get("success", False))
    return {
        "test_case_id": task_case_id,
        "grader_results": [{
            "grader_id": "gr_verify",
            "type": "code",
            "passed": success,
            "score": 1.0 if success else 0.0,
            "metadata": {"error_message": exec_result.get("error_message")},
        }],
        "metadata": {
            "agent_execution_time": meta.get("agent_execution_time"),
            "token_usage": meta.get("token_usage", {}),
            "turn_count": meta.get("turn_count"),
            "actual_model_name": meta.get("actual_model_name"),
        },
    }

I'd write this as a standalone mcpmark-openeval-adapter package (same shape as adapters/autogen-openeval-adapter in the EvalPort repo — to_openeval()/from_openeval(), tests against the real validator, doesn't touch MCPMark's code at all) rather than a PR into this repo, since it only needs the public meta.json/tasks/ shapes and MCPMark already has enough on its plate maintaining 127 verified tasks across 5 services.

Two questions before I build it:

1. Is meta.json's shape (the fields above) stable enough across the run-* layout that I should treat it as a de facto public interface, or does it change often enough that I'd be chasing it?
2. Any interest in a converter going the other direction eventually — importing an EvalPort suite as description.md/verify.py scaffolding for a new MCPMark task? That's a bigger ask (verify.py is Python-executed against live sandboxes, not a declarative grader spec), so I'm not proposing it now, just checking if it's worth designing for.

Happy to build the adapter regardless and link it back here once it exists — mainly want to sanity-check the meta.json stability question first since that's the part I can't verify from the code alone.

— Sahi, independent contributor (not affiliated with this project)