Restructure the system prompt into a documented shape #2364

Activity

1. sentry-junior commented on Oct 29, 2026

   

   sentry-juniorBot

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Research results (all sources fetched 2026-10-29). This answers the open research items in the issue. Unverified items are marked.

   Sources agree on these points

   • Put a short role or identity first. Put rules that apply in every turn before rules about the current situation. Put context that changes per turn last.
     • OpenAI's general guide: Identity → Instructions → Examples → Context (prompt-engineering). The GPT-5.5/5.6 guides: Role → Personality → Goal → Success criteria → Constraints → Tools → Output → Stop rules (5.6). The GPT-6 guide gives no order. It says GPT-6 is "more sensitive to information in context" (gpt-6-astra).
     • Anthropic gives no fixed order. Use labeled sections. Put long data first and the query last (best practices, context engineering).
   • Keep personality short and apart from behavior rules. OpenAI 5.5/5.6 keep separate Personality and Collaboration style blocks: "Neither should replace clear goals, success criteria, tool rules, or stopping conditions." Claude Code output styles are a layer that sits on top of fixed task behavior.
   • Have one explicit evidence and honesty rule.
     • OpenAI Model Spec: #avoid_errors, #express_uncertainty, and #avoid_sycophancy ("should not change its stance solely to agree with the user").
     • OpenAI 5.6: "report missing evidence instead of guessing"; "Absence of evidence should not automatically become a factual 'no.'"
     • Anthropic: "audit each claim against a tool result from this session". Anthropic says this "nearly eliminated fabricated status reports" (Fable 5 guide). Also: allow "I don't know" (reduce hallucinations).
   • Treat tool output, documents, and memory as information with no authority.
     • OpenAI Model Spec #chain_of_command, #ignore_untrusted_data (spec).
     • Anthropic: retrieved or tool text in a system message "gives that text operator-level authority" (system messages). CLAUDE.md is a user message and counts as context (memory).
   • State each rule once. OpenAI 5.6: conflicting rules "create more instability than missing detail". Leaner prompts gave about 10–15% better eval scores with 41–66% fewer tokens; OpenAI calls this directional. Anthropic: drop "CRITICAL/MUST", and put tool guidance in the tool descriptions (building effective agents).
   • Allow per-model variants. Codex keeps one template per model. Claude Code switches per-model lean and leanRewrite variants on with flags.

   Claude Code 2.1.312 assembly order (taken from the linux-x64 binary, function rsr)

   • Each section has a scope. "Shared" sections are the same every session, so they can be cached. "Session" sections can change. __SYSTEM_PROMPT_DYNAMIC_BOUNDARY__ goes between the shared and the session sections.
   • Shared sections, in default mode: intro → System → Doing tasks → Executing actions with care → Using your tools → Tone and style.
   • Session sections: about 29 conditional ones. They include Communicating with the user, auto memory, Environment (model IDs only), Messages from the system, Delivering work, Corrections, Writing for the user.
   • The system prompt holds no context. CLAUDE.md, environment, git status, and the date arrive as <system-reminder> attachments in user messages. Unverified: that these go on the first user turn.

   Junior transport (at a8ddb35)

   • Each turn is one Anthropic system block plus user messages. No message uses a developer role. OpenAI models go through the gateway's anthropic-messages API (pi/client.ts). The gateway docs do not say how system maps to OpenAI roles. Unverified.
   • Runtime context, AGENTS.md, and thread context already go in a user-role message. This matches both vendors. But the <runtime-turn-context> header calls the block "trusted runtime facts", and that block also holds memories and plugin context.
   • <plugin-system-context> is added to the system prompt every turn (agent/prompt.ts). So plugin-written text gets operator authority. Unverified: whether it stays the same from turn to turn.
   • Pi sets cache markers on the system block, the last tool, and the last user block. The prefix order is tools → system → history → context → request. Unverified: whether cache_control has any effect for OpenAI models.
   • promptSnippet and promptGuidelines are deprecated. Junior renders them as <tool-guidance> in the user context message, and three are still in use.

   Conflict with a big-bang rewrite
   OpenAI 5.6 says: "Do not rewrite a working prompt stack all at once." It asks for "the smallest targeted instruction that fixes a measured regression." A restructure therefore needs baseline evals first. The first step moves the existing text without rewording it, and the evals must show no change. After that, each change gets measured on its own.

   Open decisions

   • Where personality goes. OpenAI puts it second. Claude Code and Anthropic put tone near the end of the static rules.
   • Whether <plugin-system-context> stays in the system prompt or moves to the context message.
   • Per-model wording variants inside one fixed block order.
2. sentry-junior commented on Oct 29, 2026

   

   sentry-juniorBot

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Follow-up research: persona placement, eval coverage, caching

   Sources fetched on 2026-10-29. Inferences are marked.

   1. Persona placement

   No vendor source tests where a persona block sits relative to the behavior rules. So any choice is an inference from the sources below.

   For the end of the static prompt

   • Claude Code 2.1.312: Tone and style is the last shared (static) section, from the binary.
   • claude.ai Fable 5.1 prompt (release notes): the full <tone_and_formatting> block is in the middle. The last block is a one-line <tone_preference>. This matches Anthropic's advice to repeat the key style rule near the end of a long prompt.
   • GPT-4.1 guide (cookbook): "If there are conflicting instructions, GPT-4.1 tends to follow the one closer to the end of the prompt." This is the only first-party statement that position changes which rule wins.

   For an early position

   • OpenAI GPT-5.5 and 5.6 guides: Role → Personality → Goal → … Personality "controls how the assistant sounds" and should not "compensate for unclear goals".
   • claude.ai Sonnet 5.5 and Haiku 5.5 prompts: tone sits in the middle, with no reminder at the end.

   Risk of the end position. By the GPT-4.1 finding, the last text has an advantage in a conflict. Today SOUL.md holds behavior rules too: formatting defaults, reaction rules, "don't claim abilities", and epistemics. At the end, text that the operator wrote could win over core rules.

   Proposal (fits the current lean toward the end):

   • Put the persona last in the static prompt, before World.
   • Before that, move every behavior rule out of SOUL.md, so the persona block is tone only.
   • Keep the precedence line "Platform mechanics and output rules override personality and world context when they conflict." Haiku 5.5's guide uses the same pattern: "The rules in this system prompt hold for the whole conversation."
   • Add eval cases where a strong persona conflicts with a core rule (see §2). These cases decide whether the end position is safe.

   2. Eval coverage for prompt changes

   The current setup can catch a prompt change that breaks things. It cannot reliably show if a restructured prompt is better or worse.

   What is real. Behavioral and integration evals share src/fixture/agent.ts. The agent, model, router, Guardian and compaction are real. The app is built with createApp(), so the real buildSystemPrompt runs. The two suites differ only in policy: integration fails on any miss, and behavioral uses rubric scores with an 80% pass floor.

   Gaps:

   • No persona and no world. Inferred from resolveHomeDir() in chat/discovery.ts: there is no app/ dir near packages/junior-evals. So loadSoul() falls back to DEFAULT_SOUL (prompt.ts:43), and there is no World section. Blocks 7 and 8 are never tested. Not confirmed with a run.
   • prompt.ts triggers nothing. Behavioral and integration path filters only watch packages/junior-evals/**. A prompt PR needs a trigger-evals* label. No workflow runs on push, schedule or workflow_dispatch, so there is no baseline for main.
   • No profile pinning. The router picks the profile. Defaults: standard = openai/gpt-6-luna, handoff = anthropic/claude-opus-5.5. Coverage of each model depends on the router.
   • One sample per case. There is no repeat count. src/recording-setup.ts keeps the recordings of a passed test and drops a failed one. So a rerun is a retry until green, and pass rates lean optimistic.
   • No A/B. Nothing compares per-case scores between the base and head prompt. scripts/report.mjs uploads runs to evals.sentry.dev when SENTRY_EVALS_API_KEY is set (not checked).
   • The judge sees only reply text (policies/evals.md). "Claims a cause without a tool call" needs an assertion on conversation.toolCalls, not a rubric.

   Coverage by block:

   Block	Coverage	Gap
   1 Identity	conversation/actors, participation	thin, OK
   2a Infer intent	participation (8 cases), coding/github	—
   2b Evidence and claims	skills/providers "double-check a source-backed fact", integration/conversation/attachments	No case where a provider tool fails (404/5xx) and Junior must diagnose the cause.
   2c Corrections	google/scheduling (correction with new info), skills/action-review	No pushback without new evidence.
   3 Trust model	actors, event-automations/delivery (provider text as data), Guardian suite	No stale or wrong memory against tool evidence. No injection through memory, AGENTS.md, plugin context or tool results in the main agent.
   4 Actions and safety	action-review, guardian/, scheduler credentials	good
   5 Cross-tool rules	coding/subscriptions, skills/routing	OK
   6 Output	conversation/output, steering	No check that a reply states uncertainty.
   7 Persona	none	missing
   8 World	none	missing
   Automation and child agent	scheduler/*, agent/delegation, integration/conversation/subagents	OK

   Needed for a prompt eval setup:

   1. Cases for the target failure: a tool history with a failed 404, then pushback with no new evidence, plus a recalled memory with a plausible wrong cause (insertMemory()). Assert a new tool call or an explicit "unverified". Add variants for a provider 5xx, stale memory against fresh tool output, and injection in memory, plugin context and tool output.
   2. A full SOUL/WORLD fixture (for example a copy of junior-prod's), plus persona-against-rule conflict cases.
   3. An option to pin standard or handoff per run, and run both for prompt PRs.
   4. Add packages/junior/src/chat/prompt.ts and SOUL/WORLD loading to the behavioral and integration path filters. Add a run on main as a baseline.
   5. N repeats per case for prompt PRs, and a per-case score diff between base and head.

   3. Caching

   The policy will state two rules: the system prompt is static, and earlier transcript turns never change.

   • The proposed shape keeps all per-turn content in the context message, after the static prompt. It fits both rules.
   • <plugin-system-context> is the one item that the runtime builds on each turn and adds to the system prompt (agent/prompt.ts:571). Verified: the systemPrompt hook gets only the plugin's base context and platform, with no turn data (systemPromptPluginContext, plugins/agent-hooks.ts:172). No plugin in getsentry/junior or junior-prod implements it today. So it is static in practice. But the contract does not enforce it, because the hook is async and runs on each turn. The policy should say that the hook must return the same text for each deployment and platform.
   • The persona is static per deployment, so it can be cached in any position before the context message. The Pierre line in SOUL.md is static text, so it does not break caching.
3. sentry-junior

   added a commit that references this issue on Oct 29, 2026

   docs(prompt): State the intent of each system prompt section

   ...

   9d3e017
4. dcramer

   added a commit that references this issue on Oct 29, 2026

   test(evals): Add evals for provider failure, pushback, stale memory, …

   ...

   14973fa
5. sentry-junior commented on Oct 30, 2026

   

   sentry-juniorBot

   on Oct 30, 2026

   ContributorAuthor

   More actions

   #2371 merged. It adds the written prompt shape (policies/system-prompt.md) and reorganizes the prompt to match it. #2370, #2377 and #2410 added and fixed the core-behavior evals.

   Follow-up work is tracked here:

   • Move single-tool rules from the system prompt into tool descriptions #2411: move single-tool rules into tool descriptions.
   • getsentry/junior-prod#576: make SOUL.md voice-only.
   • Run system prompt evals with a real persona and a pinned profile #2412: run system prompt evals with a real persona and a pinned profile.
   • Fix behavioral eval cases that fail on unrelated PRs #2413: fix behavioral eval cases that fail on unrelated PRs.
   • Record which account made each MCP call #2414: record which account made each MCP call.
   • Stop using the literal name "Junior" as the Briefs attribution token #2415: stop using the literal name "Junior" as the Briefs attribution token.

   Closing this issue as done.
6. sentry-junior commented on Oct 30, 2026

   

   sentry-juniorBot

   on Oct 30, 2026

   ContributorAuthor

   More actions

   Closing as done. #2371 merged the documented prompt shape (policies/system-prompt.md) and the reorganized prompt, and #2370, #2377 and #2410 added and fixed the core-behavior evals. Remaining work is tracked in #2411, #2412, #2413, #2414, #2415, getsentry/junior-prod#576, and #2416 (merge conflicts on idle PRs).
7. sentry-junior

   closed this as completedon Oct 30, 2026