Skip to content

feat: send images to serverless agents from the playground and CLI - #822

Open
devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
feat/agent-image-attachments
Open

devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
feat/agent-image-attachments

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This adds the client side of image turns. The backend support is in diggerhq/blue feat/turn-image-attachments, which adds attachments to POST /sessions/:id/turns. That PR needs to be deployed before the images reach the agent.

attachments?: { type: "image"; mediaType: string; data: string /* base64 */; name?: string }[]

Web playground (managed-agents/Detail.tsx, images.ts):

  • The composer can take images from a picker button, a paste, or a drag and drop. Each image shows as a thumbnail you can remove, and sent images also appear in the user's message bubble.
  • imageRefusal rejects anything that isn't PNG, JPEG, GIF or WebP, anything over 5 MB, and more than 4 images, with a message shown inline.
  • A message can be sent with only an image. Queue, steer and interrupt still work for text; a message with images is queued.
  • Images travel as AI SDK FileUIPart data URLs. ManagedAgentChatTransport turns them into attachments for runManagedAgent and continueManagedAgentSession. When a session is already active, admitManagedAgentInput sends them directly.

CLI:

opencomputer run <agent> [prompt] --image ./shot.png [--image ...]
opencomputer session send <session-id> [prompt] --image ./shot.png
  • readTurnImages checks that each path is a regular file of at most 5 MB, finds the type from the file's magic bytes (the file extension is ignored), and allows at most 4 images.
  • When there are images, the prompt is optional.
  • createTurn(sessionId, input, key, attachments = []) only adds attachments to the request body when there are some, so text-only requests are sent exactly as before.

Verified locally: cli npm test (135 passed) and tsc; web vitest (299 passed), npm run build, and eslint on the changed files. The only remaining lint error in Detail.tsx (react-refresh/only-export-components on PROJECT_DETAIL_TABS) is already on main.

Link to Devin session: https://app.devin.ai/sessions/af9f2258a9af4cad96662ae2c820b4f1
Open in Devin Desktop: https://app.devin.ai/desktop/session/af9f2258a9af4cad96662ae2c820b4f1?variant=devin
Requested by: @motatoes

Polylane reviews this pull request when you ask:

Review with Polylane

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Tested locally against a real Blue worker (https://github.com/diggerhq/blue/pull/133) and a real OpenCode/model runtime, with no API mocking.

Passed: picker preview and removal; image + text, where the base64 sent matched the original file byte for byte and the model correctly identified the orange triangle and blue circle in the fixture; image-only send via paste; drag and drop; inline notices for SVG, files over 5 MB and a 5th image; image + steer gets queued; a text-only request has no attachments key. The backend returned the expected 400 codes for an empty turn, invalid bytes, 5 images, an oversized image, and data passed with a data: prefix.

Real model recognition Image-only response
Image recognized Image-only response

Follow-ups from the run:

  • A .txt renamed to .png only failed after Send, with a backend 400. Fixed in 121e8ee: the client now checks the file's bytes and shows the inline notice before sending.
  • Image thumbnails don't show in history after a reload. The backend deliberately deletes image bytes once a turn finishes, so history only has the image name and hash.
  • Turns admitted while a run is active only appear in the chat after a reload. A text-only control showed the same thing, so this was already the case and isn't caused by images.

Not covered: workerd forwarding (covered by an integration test instead), sending an image successfully from the CLI (a local route mismatch between /api/managed-agents and /v1 blocked it; the CLI's rejection of a non-image file was checked and passed), and Slack.

…chments

Co-Authored-By: Mohamed <mo@digger.dev>

# Conflicts:
#	web/src/managed-agents/Detail.tsx
…chments

Co-Authored-By: Mohamed <mo@digger.dev>

# Conflicts:
#	web/src/managed-agents/Detail.tsx

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant