Skip to content

Expose cursor position and modifier keys for computer actions - #136

Merged
NikhilShahi merged 8 commits into
mainfrom
devin/1791410498-computer-action-primitives
Oct 8, 2026
Merged

NikhilShahi merged 8 commits into
mainfrom
devin/1791410498-computer-action-primitives

Conversation

@NikhilShahi

@NikhilShahi NikhilShahi commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Expose desktop cursor position and pointer modifier keys through existing browserctrl functionality. Keep the existing coordinate-based scroll API intact, with a separate helper for scrolling at the current pointer and the standard ComputerActionResponse for every action.

Changes

  • Add sync/async cursor_position() returning the standard ComputerActionResponse through the shared request dispatcher; no dedicated response model or generic response-model transport helper.
  • Add ComputerAction.CURSOR_POSITION and its request/coordinate variants to the existing general action and response unions.
  • Include ScrollAtCursorActionParams in both TypedDict and legacy model ComputerActionParams unions; sync/async scroll_at_cursor() uses the shared request dispatcher, which validates both scroll payload shapes.
  • Preserve required coordinates and deltas on scroll() and its request models/TypedDicts; use scroll_at_cursor() for coordinate-free scrolling. Python's existing coordinate-optional click() behavior is unchanged.
  • Append optional modifier keys to click, drag and scroll, preserving existing positional arguments and no-modifier wire payloads.
  • Add sync/async, model/dictionary wire parity, error, default-delta and typing regression coverage.
  • Minor-bump 1.9.4 → 1.10.0; no backend changes or new dependencies.

Compatibility

Existing click/scroll signatures and coordinate-based ScrollActionParams fields are preserved. The main request unions also accept coordinate-free scroll payloads; generic scroll-request consumers must check for coordinates before reading them. Incomplete or null coordinates remain rejected. The general action enum, request union and response-data union gain cursor position, following prior SDK expansion practice: strict exhaustive switches/maps and generic response consumers may need to handle the new variant. All actions return ComputerActionResponse; callers use isinstance(data, ComputerActionResponseDataCursorPosition) before reading cursor coordinates.

Validation

  • poetry run ruff check .; changed-file ruff format --check; git diff --check
  • poetry run pytest tests --ignore=tests/sandbox/e2e -q: 632 passed, including mypy and Pyright fixtures
  • Sync/async model/dictionary transport parity and mypy/Pyright fixtures cover current-pointer requests through the shared dispatcher.
  • poetry check --lock; poetry build (1.10.0)
  • Earlier disposable live sessions verified cursor reads, screenshots, modifiers and current-pointer actions; live checks were not rerun for this API refinement.

Note

Low Risk
SDK-only API expansion with backward-compatible signatures; consumers with exhaustive action/response unions may need new variants.

Overview
Adds cursor position and scroll-at-cursor computer actions on sync/async client.computer_action, with new request/response types (CursorPositionActionParams, ScrollAtCursorActionParams, ComputerActionResponseDataCursorPosition) wired through the shared action dispatcher. Coordinate-based scroll() stays required for x/y; dict payloads without coordinates resolve to scroll-at-cursor.

Optional modifier keys are added to click, drag, and both scroll helpers (omitted from the wire when unset). Bumps package version 1.9.4 → 1.10.0 with regression tests and typecheck fixtures for wire parity and invalid keys types.

Reviewed by Cursor Bugbot for commit 7591ce8. Bugbot is set up for automated code reviews on this repo. Configure here.

@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@NikhilShahi
NikhilShahi merged commit 674e14c into main Oct 8, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants