Skip to content

v0.22.0 — the failures named from outside, and the control that is absence - #23

Merged
sshlg merged 1 commit into
mainfrom
feat/mast-schema
Aug 31, 2026
Merged

v0.22.0 — the failures named from outside, and the control that is absence#23
sshlg merged 1 commit into
mainfrom
feat/mast-schema

Conversation

@sshlg

@sshlg sshlg commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

graph-engineering.md argued everything from failure and cited nothing

Why Do Multi-Agent LLM Systems Fail? (2025) measured the same ground: traces from seven mainstream frameworks — MetaGPT, ChatDev, AG2, Magentic-One — with human annotators independently analysing ~150 traces at Cohen's kappa = 0.88, yielding 14 failure modes in three groups.

The reason to carry it is that the groups land on this file's own five-field node contract without being bent to fit:

MAST group The field it lands on
System design flaws — unclear interfaces, overlapping roles, wrong tool config input/output, and owner
Inter-agent alignment failures — inconsistent objective, downstream misinterpretation, contradictory operations the edge payload (§3's Carries column)
Missing task verification — an agent reports completed and the result does not meet the requirement check, and §6's checker node

An outside measurement arriving at the same joints is worth more than another argument from failure.

And the load-bearing result is the negative one. Better prompts, more explicit roles and retries bought ChatDev 15.6% — the authors conclude the modes are architectural rather than bugs, which is exactly the claim §1 makes about the two fields nobody draws.

governance.md never named the control before per-tool authorisation

The model cannot reason about capabilities it does not know exist.

A tool absent from the schema cannot be invoked, argued for, or probed for a bypass. A tool present and refused at call time is a negotiation, and negotiations are won sometimes.

Sub-agent isolation needs both halves: schema filtering at construction and message_history = None at execution — a sub-agent handed its parent's transcript has been told about every capability you carefully removed from its schema.

Two numbers pointing opposite ways, on purpose

Eager MCP schema loading at startup 40% of the context budget before the first user message
Metadata index, schema on selection under 5%
The same system's own prompt and tool schemas built eagerly, in the constructor

The rule is not lazy is better. It is eager for what you own and always need, lazy for what is foreign and might not be used — the first removes latency and races from the hot path, the second removes a cost you cannot predict.

Approval fatigue is a safety failure, not a UX complaint

No persistence → re-approving the same operations every session → blanket auto-approval, defeating the safety system entirely — through the user rather than through a bug, so nothing in the logs looks wrong. A control that is asked too often is a control on its way to being switched off.

Evidence

Check Result
npm test rc=0
Auditor 0 GAP, 14 PASS, body 4655/4750 — unchanged
Placement both findings written entirely into references; B-129 records this body at capacity and this release spends none of it

Board B-132, B-133. Ledger MS-1 … MS-7.

🤖 Generated with Claude Code

…rol that is absence

graph-engineering.md argued every rule from failure and cited nothing. MAST
measured the same ground: 14 modes in three groups, ~150 traces across seven
frameworks at Cohen's kappa 0.88 — and the three groups land on this file's own
five-field node contract without being bent to fit. Its load-bearing result is
the negative one: patching bought ChatDev 15.6%, so the modes are architectural
rather than bugs, which is the same claim section 1 makes about owner and check.

governance.md put per-tool authorisation at the moment of invocation and never
named the stronger control one layer earlier: the model cannot reason about
capabilities it does not know exist. An absent tool cannot be argued for or
probed; a present one refused at call time is a negotiation, and negotiations
are won sometimes. Sub-agent isolation needs both halves — schema filtering at
construction AND message_history = None at execution.

Two eagerness numbers pointing opposite ways on purpose: 40% of the context
budget before the first user message under eager MCP schema loading, under 5%
with a metadata index — while the same system builds its own prompt eagerly. The
rule is eager for what you own and always need, lazy for what is foreign and
might not be used.

Approval fatigue filed as a safety failure rather than a UX complaint: no
persistence produces blanket auto-approval, through the user rather than a bug,
so nothing in the logs looks wrong.

Both written entirely into references; the body is unchanged at 4655/4750, which
B-129 records at capacity.

B-132, B-133, MS-1..7. Gate: npm test rc=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sshlg
sshlg merged commit ac2015d into main Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant