Run system prompt evals with a real persona and a pinned profile #2412

Follow-up to #2364 and #2371.

The behavioral and integration evals run the real buildSystemPrompt, but they cannot show how a prompt change affects a real install.

Gaps:

• No persona or world. resolveHomeDir() in chat/discovery.ts finds no app/ directory for packages/junior-evals. So loadSoul() returns DEFAULT_SOUL ("You are a practical and concise assistant.") and there is no World block. This is inferred from the code and not confirmed with a run. Blocks 7 and 8 of the prompt shape are never tested, and neither is a conflict between Personality and a core rule.
• No profile pinning. The turn router picks the profile for each case. No case can force standard or handoff, so coverage of each model depends on the router.

Proposal:

• Add an eval fixture app/ with a short SOUL.md and WORLD.md, like a real install. Include one persona line that conflicts with a core rule, and add a case that checks the core rule wins.
• Add a way for a case to pin a profile, so one case can run on both standard and handoff.

Out of scope: repeated live sampling. #2376 was closed because it adds too much complexity for now.

via David Cramer.

--

View Session [Sentry]