Skip to content

feat(desktop): read a window through its accessibility tree - #170

Merged
httpsphl merged 4 commits into
mainfrom
feat/desktop-read
Oct 4, 2026
Merged

httpsphl merged 4 commits into
mainfrom
feat/desktop-read

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

The second part of D1 (spec 24.5). It sits on top of #169, so this diff shows #169 too until that one merges.

  • The walk. platform::desktop::read(id) walks the window's UI Automation tree:
    • one level at a time, with FindAllBuildCache, so the properties come in the same cross-process call;
    • with COM in multithreaded mode on the calling thread;
    • keeping what a person sees: the control view, with offscreen controls left out.
  • Each control gets its kind in words, name, value (Value pattern), toggle, expand/collapse and selection state, whether it is enabled, and its RuntimeId. D2 uses the RuntimeId to find the control again to act on it.
  • Left out or flattened:
    • unnamed panes, groups and custom controls give no line; their children take their place;
    • scroll bars, title bars, tips, thumbs and separators are left out;
    • the walk stops at 3000 controls or 40 levels.
  • Password fields (IsPassword) never have their value read.
  • The text. render(controls, from) gives one control per line, for example [d2] edit "Name" = "Ana" or [d4] check box "Remember" (checked). It is indented by depth and capped at 600 lines, with from to go on. by_reference finds a control by its ref.
  • Spec 24.5 is updated.

Checks

  • cargo fmt and clippy -D warnings pass.

  • Unit tests cover rendering, password hiding and paging.

  • A real test adds a button, a field, a password field and a ticked check box to the test's own window, then reads it:

    [d1] button "Save"
    [d2] edit = "hello"
    [d3] edit (password: the owner types it, you never read or type it)
    [d4] check box "Remember" (checked)
    

    The password text is never in the output, and a closed window cannot be read.

… picture of one

The first part of D1 (spec 24.3, 24.4), in the platform layer: the
top-level windows of the owner's session (shown or minimized, with a
title, not tool windows nor cloaked ones), the program each one is from
(its executable, named by its file description; a Store app by the core
window inside its frame) and whether it runs as administrator. The list
of what is never granted (Botloft, elevated windows, terminals and
command boxes with the Run box, Task Manager, password managers and the
Credential Manager) is pure code with tests. A picture of a window comes
from PrintWindow with full content, in real pixels, cropped to the DWM
frame, shrunk to logical pixels up to 1600 x 1200 and encoded as a JPEG
(jpeg-encoder). Other systems say the desktop is not there yet.

Tested with a real window of the test's own on Windows 11: listed with
its app, pictured red behind others, minimized, closed.
The second part of D1 (spec 24.5): UI Automation walks the window a
level at a time, with the properties fetched in the same call, and keeps
the controls a person sees (the control view, nothing offscreen). Each
becomes a line with a ref, its kind, name, value and state (checked,
open, selected, disabled); unnamed panes and groups give way to what is
in them, and scroll bars, title bars, tips and separators are left out.
A password field never shows its value. Readings go on from where they
stopped past 600 lines, and the walk stops at 3000 controls or 40
levels. Each control keeps its RuntimeId to be found again to act on it.

Tested with real controls in the test's own window: a button, a field,
a password field (its text never read) and a ticked check box.
@httpsphl
httpsphl enabled auto-merge October 4, 2026 21:13
@httpsphl
httpsphl merged commit 5ebacbd into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant