Skip to content

feat(miniapps): operate live apps through semantic and DOM controls - #3549

Merged
wgqqqqq merged 11 commits into
GCWing:mainfrom
wgqqqqq:wgq/miniapp-browser-lifecycle
Oct 27, 2026
Merged

wgqqqqq merged 11 commits into
GCWing:mainfrom
wgqqqqq:wgq/miniapp-browser-lifecycle

Conversation

@wgqqqqq

@wgqqqqq wgqqqqq commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

An Agent operating a MiniApp should work on the user's current instance through MiniAppBrowser, preserve unsaved state, and verify business results. This adds source-informed semantic control plus DOM/input fallbacks, and makes that route the first choice before desktop automation.

For example, the Agent can update Regex Playground's unsaved input without reloading it, or play the existing SVG Gomoku board through application-local clicks when the app has no semantic actions.

Implementation

  • Preserve the portable host port, surface sessions, target handles, and lifecycle guards. Tabs use the Web UI runtime; independent windows use authenticated Shell IPC with view-bound, expiring, single-use request claims.
  • Prefer app-declared state/actions, then supported DOM/input controls. Tool descriptions, shared interaction guidance, and open/error responses distinguish MiniApps from ordinary web pages and native desktop apps. A missing semantic bridge alone does not justify ComputerUse.
  • Add normalized clicks within a bound element, including SVG/Canvas targets. The owning surface must advertise element-position-click-v1; older hosts reject positioned clicks before dispatch and retain ordinary element clicks. Invalid, stale, hidden, or covered positioned targets fail explicitly.
  • Keep routine operation separate from authoring. Source inspection uses the existing installed-app owner; adding controls must preserve state and must not silently rewrite, reload, or transfer the user's current app.
  • Give InitMiniApp a validated state/action scaffold. Integrate Regex Playground and its skill mirror with bounded observations and shared UI/Agent business functions; preserve legacy storage and omitted draft fields.
  • Publish MiniAppBrowser's assistant-facing output schema and complete the new operation's English, Simplified Chinese, and Traditional Chinese labels.
  • Keep capability discovery consistent across Rust, Web UI, and Playbook search. Acronym matching requires consecutive initials so long workflow prose cannot outrank unrelated capabilities by matching scattered letters.

Validation

  • Full Web UI suite: 12,224 tests across 1,085 files passed, plus full Web lint (zero errors) and TypeScript checks. Runner regression coverage verifies ordinary rerenders retain the document while changed compiled content retires the old frame and bridge.
  • Core MiniAppBrowser and product tool catalog tests: 41 passed with explicit owner features and no default features, including concrete output-schema validation independent of product-full registry selection.
  • Agent/prompt, ComputerUse, and MiniAppBrowser focused tests: 216 passed; prompt catalog contracts: 9 passed.
  • Product-control contracts and shared search acceptance corpus: 17 passed. MiniApp surface contracts: 15 passed; MiniApp domain integration contracts: 43 passed.
  • Web UI MiniApp DOM/router/business tests: 38 passed. The unchanged legacy Gomoku fixture verifies center and off-center moves, AI replies, and preservation of the current document without a semantic bridge.
  • Global search/provider tests: 23 passed. Playbook tests: 9 passed; website build passed.
  • Web checks (including TypeScript and appearance/theme contracts), i18n audit, capability generation verification/tests, changed-Rust formatting, and diff whitespace checks passed.
  • Manual local dev logs confirm two real-model Gomoku games used MiniAppBrowser without ComputerUse; the second game reached a verified winning board.

Compatibility and limits

No required persisted fields are added. New positioned input is capability-negotiated; older host payloads and unknown capability strings retain round-trip coverage. App actions use the existing host permission paths, and MiniApp-owned headless Agent runs remain restricted from MiniAppBrowser.

Host and Shell must be updated together for this PR's request-claim protocol. Restart the Host and reopen independent windows to load that runtime.

Local DOM/Shell fixtures and the manual Gomoku run do not establish Windows/Linux native behavior, independent-window end-to-end behavior, or live SSH/Docker workspaces, relay/mobile remote control, Peer connections, or Detached Dispatch. Full platform validation runs in CI.

AI-assisted implementation; focused automated validation and local manual checks completed.

@wgqqqqq wgqqqqq changed the title feat(miniapps): add MiniAppBrowser with safe surface lifecycle feat(miniapps): add source-informed semantic control of live apps Oct 26, 2026
@wgqqqqq wgqqqqq changed the title feat(miniapps): add source-informed semantic control of live apps feat(miniapps): operate live apps through semantic and DOM controls Oct 26, 2026
@wgqqqqq
wgqqqqq merged commit c8b6746 into GCWing:main Oct 27, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant