Skip to content

Let an agent embed a screenshot in its reply, at a size it picks - #320

Merged
frenchie4111 merged 2 commits into
mainfrom
agent-screenshots-in-chat
Oct 9, 2026
Merged

frenchie4111 merged 2 commits into
mainfrom
agent-screenshots-in-chat

Conversation

@frenchie4111

@frenchie4111 frenchie4111 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Tool-result screenshots already showed up in the transcript as thumbnails on the tool card (#303, #314), but an agent had no way to put one in its own prose — which is where "here's the thing I just built" belongs.

Getting the screenshot into the reply

The agent never learned where the capture landed. The MCP result is image bytes, and the path Ness spills them to was derived later, in the transcript extractor. /browser/screenshot now does the spill itself and returns the path; the bridge appends one terse line to the result. The write is content-hash keyed, so it's the same file the extractor derives from the image block — one copy, not two.

A local path in markdown couldn't render anyway. The renderer's origin is app:// (http://localhost in dev), so an absolute filesystem path resolves against the wrong root, and react-markdown blanks file:// in its default URL transform. Chat markdown now routes srcs that look like local files through the attachment-read IPC the existing thumbnails already use. Works for any image on the machine, not just screenshot_tab captures. Remote srcs keep their old behaviour, and a blanked javascript: src renders nothing.

Sizing

CommonMark has no sizing syntax and raw HTML is off in chat (rehype-raw would let any model-authored markup into the renderer), so the size rides in the places an image already carries free text. Three forms, all prior art, because agents guess differently:

form example
pipe in the alt (Obsidian) ![panel|400](path)
leading-= title (markdown-it-imsize) ![panel](path "=400x300")
fragment on the path ![panel](path#w=200)

Specs are 400, 400x300, x300, 50%, clamped to 16–2000px. A title without the = is still a tooltip; alt text that merely contains a pipe is left whole.

The fragment form is why this isn't just the first two: it previously resolved to nothing and fell through to a broken <img> that rendered as blank space — the worst outcome for someone who guessed a reasonable syntax.

Unsized default is 16rem tall / 32rem wide (was 24rem tall), since one screenshot was taking about a third of the chat window.

Still renders as raw text: ![](path =200x) with the spec outside the quotes. CommonMark won't parse it as an image at all, so honouring it needs a remark transform — skipped as out of proportion to three working forms.

Verification

  • src/shared/chat-image-src.test.ts — 22 cases over src resolution (posix, file:// + percent-decode, drive letters, remote/relative/unknown-ext rejections) and the three size parsers (clamping, non-size tails, tooltip preservation).
  • New control-server test pins the returned path shape and that a repeat capture of the same bytes reuses the file.
  • react-markdown wiring confirmed in node via renderToStaticMarkup: absolute paths, file://, alt pipes and = titles all arrive intact — including a pipe escaped inside a GFM table cell; javascript:alert(1) arrives blanked and renders nothing.
  • Visual, against a live transcript in the web client: the title and fragment forms both render at the requested 200px, and the unsized control dropped from ~535×340 to ~357×218.
  • npm run typecheck, npx electron-vite build, and all 1745 tests pass.

🤖 Generated with Claude Code

frenchie4111 and others added 2 commits October 9, 2026 11:13
Tool-result screenshots already showed up in the transcript as
thumbnails on the tool card (#303, #314), but an agent had no way to
put one in its own prose — which is where "here's the thing I just
built" belongs. Two things were missing.

The agent never learned where the capture landed: the MCP result is
image bytes, and the path Ness spills them to was derived later, in the
transcript extractor. /browser/screenshot now does the spill itself and
returns the path. The write is content-hash keyed, so it's the same
file the extractor derives from the image block — one copy, not two.

And a local path in markdown couldn't render anyway. The renderer's
origin is app:// (http://localhost in dev), so an absolute filesystem
path resolves against the wrong root, and react-markdown blanks file://
in its default URL transform. Chat markdown now routes srcs that look
like local files through the attachment-read IPC that the existing
thumbnails use, at full column width. Remote srcs keep their old
behaviour, and a blanked javascript: src renders nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An agent that embeds a screenshot had no way to say how big it should
be, and the unsized default — a 24rem-tall cap — gave one screenshot
about a third of the chat window.

CommonMark has no sizing syntax and raw HTML is off in chat (rehype-raw
would let any model-authored markup into the renderer), so the size
rides in the places an image already carries free text. Three forms,
all prior art, because agents guess differently: a pipe in the alt
(`![panel|400](…)`, Obsidian), a leading-`=` title (`![x](… "=400x300")`,
markdown-it-imsize), and a fragment on the path (`…png#w=200`). Specs
are `400`, `400x300`, `x300`, `50%`, clamped to 16-2000px. A title
without the `=` is still a tooltip, and alt text that merely contains a
pipe is left whole.

The fragment form is the reason this isn't just the first two: it
previously resolved to nothing and fell through to a broken <img> that
rendered as blank space — the worst available outcome for someone who
guessed a reasonable syntax.

Default for an unsized embed is now 16rem tall / 32rem wide, down from
24rem tall, with the loading placeholder following it down.

Verified against a live transcript in the web client: the title and
fragment forms both render at the requested 200px, and the unsized
control dropped from ~535x340 to ~357x218.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@frenchie4111 frenchie4111 changed the title Let an agent embed a screenshot in its reply Let an agent embed a screenshot in its reply, at a size it picks Oct 9, 2026
@frenchie4111
frenchie4111 merged commit 822fe8b into main Oct 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant