Skip to content

--load summary misreports failed calls: failures are mixed into latency/throughput, and tool-error messages are never shown #2836

Closed

Description

@CloneOfAlex

Which version line?

v2 — current (@modelcontextprotocol/inspector@latest)

Which client?

All / shared core

Inspector version

2.13.0 (v2/main @ d7f6cee, built from source)

Node version

v22.23.4

Operating system (and browser, for the web client)

Linux (Debian)

Transport

Streamable HTTP

MCP server under inspection

A small instrumented Streamable HTTP server built on the repo's own createTestServerHttp (test-servers), exposing:

  • sleep {ms} — waits ms, returns ok
  • flaky {ms, failEvery, mode} — every failEvery-th call fails instantly (mode: "tool" → isError: true), the rest wait ms and succeed
  • unique {ms} — waits ms, then throws

It also records server-side arrivals, in-flight count and arrival rate, so the CLI summary could be checked against ground truth. Happy to share the ~100-line script.

Steps to reproduce

A — latency. mcp-inspector --cli http://127.0.0.1:7400/mcp --transport http --method tools/call --tool-name flaky --tool-arg ms=200 --tool-arg failEvery=2 --load --load-requests 200 --load-concurrency 10 --load-max-p99 250 --format json
(every 2nd call fails instantly; the successes take 200 ms)

B — messages. mcp-inspector --cli http://127.0.0.1:7400/mcp --transport http --method tools/call --tool-name unique --load --load-requests 50 --load-concurrency 5 --format json
(every call's handler throws; the SDK returns it as an isError result)

Expected behavior

  • latencyMs describes successful calls (failures reported separately, e.g. errorLatencyMs), and there is a success-only throughput.
  • --load-max-p99 judges successful calls and fails when none succeeded.
  • Tool-error results contribute their message to the summary, like thrown errors do.

Actual behavior

A. ok: 100, toolErrors: 100, latency p50: 32.21 ms — no successful call was that fast (all took ≥200 ms). The run exited 0 with --load-max-p99 250 despite a 50% failure rate. runLoad pushes a latency for every call, success or failure, and throughput counts every completed call, so instant failures make a broken server look faster (see also the dead-server case in the companion issue).

B. toolErrors: 50, errorMessages: []. Only thrown errors reach the ErrorTally; an isError result is counted with no message, so a run that is 100% rate limited (or any tool-level failure) gives no hint why.

Logs, errors, or screenshots

Suggested fix: separate success/failure latency; add okThroughput; base --load-max-p99 on successes; tally the first text block of each isError result (trimmed, capped like thrown messages) into toolErrorMessages.

Already prototyped a fix?

No response

Before you submit

  • I searched existing issues and this is not a duplicate.
  • This is not a security vulnerability report (those go through the private advisory process).

Activity

  1. cliffhall commented on Oct 29, 2026

    @cliffhall
    Member

    Triage: Priority Medium (total 7)

    • Severity 4 — --load reports something false about the server under test: failures skew latency/throughput and --load-max-p99 passes a 50%-failing run
    • Urgency 2 — wanted eventually, nothing blocked on it
    • Bonuses: +1 bug label (no milestone, assignee, epic, engagement or reporter-set priority)

    Board: #28, Status Incoming (awaiting maintainer review — no milestone yet).

  2. added this to the v2.14.0 milestone on Oct 29, 2026
  3. self-assigned this
    on Oct 29, 2026
  4. cliffhall commented on Oct 29, 2026

    @cliffhall
    Member

    Fixed by #2855, merged into v2/main; verified together with the rest of v2.14.0 in the rollup #2858.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingv2Issues and PRs for v2

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions