test(android): fix E2E animation-timeout with offline WebView spinner app - #156
Closed
pr4bh4sh wants to merge 68 commits into
Closed
test(android): fix E2E animation-timeout with offline WebView spinner app#156pr4bh4sh wants to merge 68 commits into
pr4bh4sh wants to merge 68 commits into
Conversation
On Android 13+, dumpsys window InputMethod contains both mFrame= (full InputMethod window starting at status bar y=84) and touchable region= (actual keyboard area). The code checked mFrame first, causing every element to appear covered by the keyboard. Swap strategy order to prefer touchable region on Android 13+, falling back to mFrame for Android <=12.
Introduce a session-based HTTP server that bridges JSON requests to core.Driver, enabling programmatic test execution without YAML flow files. - Add server package (pkg/server) with session management, endpoints for execute, screenshot, source hierarchy, device info, and graceful shutdown - Add `server` CLI subcommand with configurable port and signal handling - Add JSON unmarshaling for flow steps (pkg/flow/json.go) mirroring the existing YAML parser - Add json struct tags to all step types and Selector for serialization - Add server and JSON unmarshaling tests - Update documentation to reflect new server capabilities and usage instructions
…sions The previous fix required isOnScreen=true to use touchable region, but SDK 30 may lack the isOnScreen field entirely, falling back to mFrame which returns the full InputMethod window bounds. Now check negative signals (isOnScreen=false, mViewVisibility=0x8) to reject hidden keyboards, then always prefer touchable region over mFrame.
Verified against AOSP source (Android 10, 11, 13) that isOnScreen=, mViewVisibility=, touchable region=, and mGivenContentInsets= are present on all versions. The parsing now uses a 3-strategy approach: 1. touchable region (most accurate, used by stock keyboards) 2. mFrame + mGivenContentInsets (for vendor keyboards like Samsung/Xiaomi that don't set touchable insets — content insets reveal keyboard top) 3. mFrame alone with 60% height sanity check (rejects full-screen InputMethod windows that would cause false positives) Negative signals (isOnScreen=false, mViewVisibility=0x8) bail early.
- Add t.Parallel() to slow/timeout-dependent tests across all drivers (uiautomator2, wda, appium, devicelab, flutter) to speed up test runs - Extract EmulatorStarter interface in cli/test.go to decouple handleEmulatorStartup from concrete emulator.Manager, enabling proper unit testing with a mockEmulatorStarter - Fix findScrollableElement in uiautomator2 to return early when page source has elements but none are scrollable, avoiding unnecessary waits - Update airplane mode tests to match new "cmd connectivity airplane-mode" implementation (Android 11+) instead of legacy "settings put" commands - Provide mock sourceData for swipe tests that rely on page source parsing - Set explicit short FindTimeout on tests that previously used defaults, preventing slow failures in relative selector and element-not-found tests
…tion fix - Add 'sleep' command support (YAML and JSON) with FlowRunner execution for both top-level and nested steps - Rework hideKeyboard to try multiple strategies (Appium → ESCAPE → BACK) with optional 'approach' field to force a specific method - Add isInputShown() using 'dumpsys input_method' for reliable keyboard detection on SDK 36+ where 'dumpsys window InputMethod' no longer reports mInputShown - Fix hideKeyboard JSON unmarshaling (was missing json.Unmarshal call) - Register StepSleep in YAML parser with scalar (ms) and mapping syntax
…hangelog - Add isKeyboardVisible step type for querying soft keyboard state, returning boolean in CommandResult.Data - Rename hideKeyboard 'approach' field to 'strategy' for consistency; accept 'esc' as alias for 'escape' - Expose KeyboardVisible in StateSnapshot via GetState() - Add IsKeyboardVisible() public method on uiautomator2 Driver - Update CHANGELOG with sleep, isKeyboardVisible, and hideKeyboard strategy entries
Add client/python package with: - MaestroClient with session management and context manager support - Command methods: tap, input_text, swipe, scroll, assert_visible, launch_app, screenshot, device_info, etc. - Typed models (DeviceInfo, CommandResult, StepResult) with dataclasses - Custom exceptions (SessionError, StepError, ConnectionError) - Unit tests with mock HTTP server (test_client.py, test_models.py) - E2E test examples (test_e2e_android.py, test_add_contact.py) - Page object pattern example (tests/pages/) - .gitignore for venv, __pycache__, dist, .pytest_cache
Add client/typescript package with: - MaestroClient class with session management and auto-cleanup - Command methods: tap, inputText, swipe, scroll, assertVisible, launchApp, screenshot, deviceInfo, hideKeyboard, sleep, etc. - Typed models (DeviceInfo, CommandResult, StepResult) with interfaces - Custom exceptions (SessionError, StepError, ConnectionError) - E2E test example with Jest (test_add_contact.test.ts) - Page object pattern example (tests/pages/) - Jest config, tsconfig, and .gitignore for node_modules/dist
…DEVELOPER.md - Add eslint.config.mjs (ESLint v9 flat config) with typescript-eslint rules: consistent-type-imports, no-explicit-any (warn in src, off in tests), no-console (warn in src), strict equality, curly braces - Fix type-only imports in client.ts and commands.ts per lint rules - Fix curly brace style in models.ts (containsChild assignment) - Add lint and lint:fix npm scripts - Add DEVELOPER.md with project structure, build/test/lint instructions, code conventions, and guide for adding new commands
…ce test - Rework conftest.py to support pytest-xdist: each worker gets its own maestro-runner server on a unique port targeting a specific device via adb device discovery and ANDROID_SERIAL assignment - Add _discover_devices() and _worker_index() helpers for parallel setup - maestro_server fixture now yields (url, device_serial) tuple - client fixture passes deviceId capability when running in parallel - Add test_contact_persists.py: POM-based test that creates a contact, cold-relaunches the app, and verifies the contact persists - Update README with parallel test instructions and env var reference
…ut protection - Add Info logging across all 7 CDP connection steps (detect, source, connection, forward, websocket, browser, ready) plus disconnect and cleanup - Add document.visibilityState check before CDP element finding — skip CDP entirely when WebView is not visible (background tab), eliminating wasted round-trips - Add 3s timeout on all Rod/CDP calls to prevent indefinite hangs when WebView is suspended or unresponsive - Add automatic cleanup (Rod browser, ADB forward, local socket) when dead connection is detected during element finding - Collapse AX tree text search from 12 CDP round-trips to 1 call with Go-side role prioritization
…ct persistence tests TypeScript: - Rework tests/setup.ts to support Jest parallel workers: each worker gets its own maestro-runner server on a unique port with a specific device assigned via adb discovery and JEST_WORKER_ID - Add jest-html-reporters and jest-junit for HTML and JUnit XML reports - Configure reporters in jest.config.js (output to ./reports/) - Add test_contact_persists.test.ts: POM-based test that creates a contact, cold-relaunches the app, and verifies persistence Python: - Add pytest-html to dev dependencies for HTML report generation - Configure pytest addopts for HTML and JUnit XML report output - Add setuptools package discovery config for maestro_runner
Previously, on every launchApp the Flutter connection state was reset unconditionally, causing repeated discovery attempts for apps that were already determined to not be Flutter. Now only resets if a connection was previously established. Also moves attempted=true to the top of reconnect methods so failures don't retry every step.
Rename jsHelperCode to JSHelperCode so the devicelab driver can inject the same element-finding helpers into WebView pages.
Add ForwardToAbstractSocket, ForwardTCPToAbstractSocket, RemoveTCPForward, and CDPSocketPath to support forwarding CDP traffic from local sockets to on-device WebView debug sockets.
Introduce core.Element interface with Text, Input, and Clear methods. NativeElement wraps UIAutomator2 elements, WebElement wraps Rod/CDP elements — allowing DeviceLab commands to interact with both uniformly.
… CDP Replace direct ActiveElement/SendKeys calls with findFocused() returning a core.Element, enabling unified text input for native and web elements. Add ensureWebViewConnection() to tapOn for automatic WebView detection. Wire SetWebViewForwarder and Close in driver setup.
Bundle the on-device instrumentation agent (Android) and WebDriverAgent replacement (iOS) source projects for building the DeviceLab native drivers.
…ELOPER.md - pyproject.toml: add ruff >=0.4.0, mypy >=1.10, types-requests dev deps; configure ruff (line-length=100, E/W/F/I/B/UP/N/S/RUF rule sets, S10x ignored in tests) and mypy (strict mode on maestro_runner/) - Makefile: add lint-py (ruff check + mypy) and lint-py-fix targets - DEVELOPER.md: new file documenting setup, lint, test, and code conventions - __init__.py: sort __all__ entries alphabetically; reorder imports - client.py: wrap long lines (>100 chars) to satisfy E501 - commands.py: explicit str() cast for selector text; wrap long dict literal - conftest.py: use collections.abc.Generator (UP035); fix blank-line isort - contact_list_page.py / edit_contact_page.py: fix import ordering (I001) - test_add_contact.py / test_contact_persists.py: fix blank-line import order - test_client.py: remove trailing blank line before import block - test_e2e_android.py: add timeout=30 to requests.post (S113); fix imports - test_models.py: remove unused pytest import (F401); remove blank line
- Add pushy descriptions to combat undertriggering (Make sure to use...) - Add explicit negative triggers (DO NOT use for) in description - Add allowed-tools and metadata fields (author, version, category, tags) - Move all when-to-use info to description field (not body) - Replace CRITICAL headers with explain-the-why rationale - Add step-numbered workflow sections for explicit ordering - Add Do NOT use section in body for quick disambiguation
UIA2 Keyboard Tests: - Add mInputShown=true gating to keyboard visibility shell mock responses in keyboard_test.go, driver_test.go, and commands_test.go - Keyboard bounds/visibility now properly verified only when IME input is active on device CLI Parallel Mode Tests: - Introduce autoDetectDevicesFn function variable for test injection - Override in TestDetermineExecutionMode_ParallelWithoutAutoStart to avoid environment-sensitive adb device detection - Ensures CI/test determinism independent of emulator/device state Keyboard Hide Command Tests: - Enhance MockShellExecutor with sequential responses (pop-from-slice) - TestHideKeyboardSuccess now provides pre/post visibility states - Handles pre-check (mInputShown=true) and post-check (mInputShown=false) Python Skill Documentation: - Add e2e Android test lock troubleshooting (socket-based process lock) - Document venv path requirements (client/python/.venv) - Add Reliable Full-Client Run Order section (unit → server → e2e) - Capture lessons: avoid combined pytest runs on single emulator New Skill: merge-upstream - Create .github/skills/merge-upstream/SKILL.md - Hardcode upstream URL: https://github.com/devicelab-dev/maestro-runner - Full merge workflow: fetch, merge, conflict resolution, verification - Includes troubleshooting for common merge scenarios
Add new git-commit skill with conventional commits workflow and evals. Improve descriptions for all skills to trigger more reliably: - go-test-runner: expand triggers, add find/head/tail/tee to allowed-tools - merge-upstream: expand triggers, improve stash guidance, consolidate test steps - python-test-runner: add Step 4 iOS test section, improve run-all sequence - typescript-test-runner: expand triggers and instructions Add evals/evals.json for every skill to enable performance benchmarking.
# Conflicts: # pkg/driver/browser/cdp/jshelper.go
…roid ci(workflows): add unit test jobs and Android e2e workflow
Integrating latest changes from https://github.com/devicelab-dev/maestro-runner: - Add cloud provider abstraction for test result reporting (SauceLabs support) - Fix CDP waitForPageReady crash - Update DeviceLab Android driver APK - Clean up dead code and update airplane mode tests - Update changelog and release notes for v1.1.1 Resolved merge conflicts in: - .gitignore (kept both entries) - pkg/flutter/semantics.go (removed unused regex patterns) - pkg/driver/uiautomator2/commands_test.go (kept local comments)
Move skill configurations and GitHub templates from .github/ to repository root: - Moved ISSUE_TEMPLATE, PULL_REQUEST_TEMPLATE.md, copilot-instructions.md to root - Moved workflows directory to root - Consolidated skills directory in .claude/skills/ (proper location) - Removed now-empty .github directory
- Restore fork's multi-strategy hideKeyboard (appium/escape/back) and its tests; upstream's keyboard_hide_test.go was added by the merge and targets upstream's implementation, so it is removed to keep the fork's tests green. - Make getKeyboardBounds parse 'dumpsys window InputMethod' directly and have parseKeyboardFrame treat mInputShown=false as not-shown, so the devicelab-dev#139 keyboard-blocking regression tests pass. - Keep fork's Condition.Script under the upstream 'true' yaml key (matches the fork's parser_test.go).
Unify the four screenshot-based drivers (uiautomator2, wda, devicelab, appium) behind a single core.WaitForScreenStatic implementation. All now honor step.sleepMs/threshold and return Success:false with a "Timed out" message when the screen never settles, restoring the fork's fail-on-timeout semantics and dropping appium's duplicate PNG decoder in favor of core.ImageDifference.
The shared conftest opens one session per test process via the autouse _track_client -> client fixture. test_e2e_android.py opened a second session, which the uiautomator2 device lock (/tmp/uia2-<serial>.sock) rejected with "device already in use". Reuse the conftest client session via the session_id fixture (yielding client.session_id) so only one session is active; teardown stays with the client fixture.
In the TypeScript test setup, teardown killed the maestro-runner server but left its stdout/stderr piped to serverLogStream. After the stream was ended, the exiting child flushed final output into it, throwing "write after end" and crashing the runner after tests had already passed. Unpipe the child's streams before ending the log stream.
…esting guidance Un-ignore .claude/skills so skill definitions are versioned. Extract the common device-harness error-handling (uiautomator2 single-session lock, stale /tmp/uia2-* PID guard, pkill self-match, teardown stream race) into a shared device-integration-testing skill that the Python and TypeScript test-runner skills now reference instead of duplicating. Co-Authored-By: opencode <noreply@opencode.ai>
GitHub Actions only reads workflows from .github/workflows/, so the root workflows/ files were never picked up (the historical CI runs predate today's main overwrite). Move ci.yml and e2e-android.yml into the standard location so they run. Co-Authored-By: opencode <noreply@opencode.ai>
Exercise waitForAnimationToEnd in CI: the Python settle test (test_wait_for_animation_to_end.py) and the TypeScript timeout test (test_wait_for_animation_never_ends.device.test.ts), which guard the fail-on-timeout behaviour introduced by the animation fix. Co-Authored-By: opencode <noreply@opencode.ai>
…imeout The waitForAnimationToEnd timeout test relied on an external spinner page (discuss.wxpython.org) that renders static (or is unreachable) inside the CI sandbox, so the screen was already static and the test false-failed. Serve a local CSS spinner from an HTTP server on the runner, reached by the emulator at 10.0.2.2, so the infinite animation is reliably reproduced. Also fix the TS teardown: server.close() blocks on the browser's keep-alive connection, so force-close connections first (same class of bug as setup.ts). Add the Python timeout test to the E2E workflow so both clients guard the fix. Co-Authored-By: opencode <noreply@opencode.ai>
The locally-served spinner (and the earlier external URL) both failed in CI because the GitHub-hosted emulator cannot reach the runner's loopback and Chrome won't render a local file:// from an intent. Replace them with the emulator's camera preview, a live feed that never settles and needs no network or browser. This makes the waitForAnimationToEnd timeout guard reliable in CI. Drop the now-unneeded HTTP server (and its closeAllConnections teardown fix) from the TS test. Co-Authored-By: opencode <noreply@opencode.ai>
# Conflicts: # .gitignore # pkg/flow/step.go
Add a "Language Clients" section to the README and dedicated tutorials under docs/clients/ covering setup, selectors, the Page Object Model pattern, advanced calls, and parallel execution. These clients (and the REST API server they wrap) are local additions not yet present in upstream.
* fix(lint): clear golangci-lint and ruff issues blocking CI These were pre-existing repo-wide lint failures (not introduced by the client expansion) that blocked the PR's Test Go and Test Python Client jobs: - errcheck: ignore close/remove error returns in webview.go, devicelab_ios/ setup.go, wda/runner.go, flutter/wrapper.go, simulator/recording.go - staticcheck (ST1005): drop trailing punctuation from a build.go error string - unused: remove dead waitKeyboardHidden in uiautomator2/keyboard.go - ruff (S607): noqa on subprocess.run(["adb", ...]) in the animation test The TypeScript client job and Python mypy/pytest already passed. * chore(ci): trigger workflow run for lint PR
…browser steps (#1) * feat(clients): expand typed bindings with gesture, media, device and browser steps Add typed MaestroClient methods for the most common step types not yet wrapped in the TypeScript and Python clients: - Gestures: doubleTapOn, longPressOn, dragAndDrop, scrollUntilVisible - Assertions & media: assertScreenshot, takeScreenshot, copyTextFrom, pasteText, setClipboard - AI & scripting: assertWithAI, evalScript, runScript, evalBrowserScript - Device control: setLocation, setAirplaneMode, toggleAirplaneMode, setNetworkConditions, openNotifications, setDarkMode, setOrientation - Browser (web): openBrowser, switchTab, closeTab, getConsoleLogs, clearConsoleLogs, assertNoJSErrors, mockNetwork Each method maps 1:1 to a server step type. The low-level executeStep/execute_step still forwards any raw step dict, so the remaining ~90 server step types remain reachable. Docs (client READMEs + tutorials) updated with a full method table and examples. * fix(lint): clear golangci-lint and ruff issues blocking CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the failing E2E Android (API 36) check on PR #1. The test_wait_for_animation_never_ends tests (Python + TypeScript) asserted waitForAnimationToEnd times out on an infinitely-animating screen, but they opened the camera expecting a live preview. On the CI emulator the camera preview returns byte-identical frames, so the driver saw 0.0% diff and concluded the screen was static (success=True) instead of timing out. Server logic is correct; the test premise was wrong.
Fix: replace the camera with a tiny offline native Android app (dev.maestro.animationtest) showing a full-screen WebView rendering a perpetually rotating CSS spinner via loadDataWithBaseURL (no network, no local server). A WebView's CSS animation is independent of the system animator/window/transition scales, so it keeps rotating even with disable-animations: true — unlike a native View/Animator. Tests launch the app, poll adb dumpsys window to confirm it is foreground, then call waitForAnimationToEnd, which now reliably times out.
Changes: e2e/android-animation-app/ (app source), drivers/android/animation-test-app-debug.apk (prebuilt, per existing convention), .github/workflows/e2e-android.yml (install APK), and the two never-ends test rewrites.
Verified locally on an emulator: both device tests pass; no regressions in other device tests; ruff/mypy/tsc/eslint pass. (go test has 2 environmental failures due to a connected emulator, unrelated to this change.)