Repository navigation
feat(desktop): read a window through its accessibility tree - #170
Merged
Merged
Conversation
… picture of one The first part of D1 (spec 24.3, 24.4), in the platform layer: the top-level windows of the owner's session (shown or minimized, with a title, not tool windows nor cloaked ones), the program each one is from (its executable, named by its file description; a Store app by the core window inside its frame) and whether it runs as administrator. The list of what is never granted (Botloft, elevated windows, terminals and command boxes with the Run box, Task Manager, password managers and the Credential Manager) is pure code with tests. A picture of a window comes from PrintWindow with full content, in real pixels, cropped to the DWM frame, shrunk to logical pixels up to 1600 x 1200 and encoded as a JPEG (jpeg-encoder). Other systems say the desktop is not there yet. Tested with a real window of the test's own on Windows 11: listed with its app, pictured red behind others, minimized, closed.
The second part of D1 (spec 24.5): UI Automation walks the window a level at a time, with the properties fetched in the same call, and keeps the controls a person sees (the control view, nothing offscreen). Each becomes a line with a ref, its kind, name, value and state (checked, open, selected, disabled); unnamed panes and groups give way to what is in them, and scroll bars, title bars, tips and separators are left out. A password field never shows its value. Readings go on from where they stopped past 600 lines, and the walk stops at 3000 controls or 40 levels. Each control keeps its RuntimeId to be found again to act on it. Tested with real controls in the test's own window: a button, a field, a password field (its text never read) and a ticked check box.
httpsphl
enabled auto-merge
October 4, 2026 21:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The second part of D1 (spec 24.5). It sits on top of #169, so this diff shows #169 too until that one merges.
platform::desktop::read(id)walks the window's UI Automation tree:FindAllBuildCache, so the properties come in the same cross-process call;RuntimeId. D2 uses theRuntimeIdto find the control again to act on it.IsPassword) never have their value read.render(controls, from)gives one control per line, for example[d2] edit "Name" = "Ana"or[d4] check box "Remember" (checked). It is indented by depth and capped at 600 lines, withfromto go on.by_referencefinds a control by its ref.Checks
cargo fmtandclippy -D warningspass.Unit tests cover rendering, password hiding and paging.
A real test adds a button, a field, a password field and a ticked check box to the test's own window, then reads it:
The password text is never in the output, and a closed window cannot be read.