WatchThis

English · 简体中文 · 日本語 · 한국어 · Français

WatchThis turns a screen recording or uploaded video into a reviewed, portable Computer Use skill. It learns the demonstrated workflow, confirms the intended outcome with the user, and keeps uncertain details visible instead of silently guessing.

Video alone can miss a quick click or hotkey. During a recording made in WatchThis, the application therefore aligns the video with structured interaction evidence captured on the same timeline.

At a glance

Focus	Summary
Problem	A screen recording shows visible states, but it can miss fast input and does not reliably reveal intent.
Approach	Align video with a privacy-aware interaction trace, analyze overlapping time windows, preserve uncertainty, and ask the user to review the inferred workflow.
Output	A schema-validated Skill ZIP with readable instructions, Codex metadata, and a platform-neutral workflow specification.
Current scope	A Windows-first MVP with type checking, unit tests, and targeted GUI smoke scripts. Real model endpoints and every generated workflow still require environment-specific validation.

Download

• WatchThis 0.3.2 Windows x64 installer
• Release notes and all assets
• SHA-256 checksum file

Installer SHA-256: 65902A849D08A55271162603CE27E8E0EC1437B7342D31BBD457A8301A1ACF5F

The packaged build includes the self-contained .NET capture helper, so judges do not need to rebuild the application or install the .NET SDK. A supported model endpoint is still required for real analysis.

How it works

1. Record a demonstration in WatchThis, or select an existing video.
2. When recording live, collect a synchronized operation trace in addition to the screen video.
3. Analyze the demonstration in logical windows of at most 60 seconds, with overlap at window boundaries.
4. Merge the observations into a structured workflow while preserving evidence gaps as uncertainty.
5. Ask the user to confirm the goal, output format, rules, and any details that could change the result.
6. Save the confirmed workflow in the Skill library, export it as a ZIP, or install it into the Codex personal skill directory.

Generated skills describe how an agent should operate a visible computer. They do not provide Computer Use by themselves.

Live recording  -> video + privacy-aware SessionTrace --+
Uploaded video  -> video (+ matching trace, if any) -----+
                                                        |
                                                        v
       overlapping windows (<= 60 s) -> aligned multimodal analysis
       -> ObservationSegments -> merged WorkflowSpec -> user review
       -> schema-validated Skill ZIP -> export or Codex installation

The detailed process boundaries and evidence rules are documented in docs/architecture.md.

Built with Codex and GPT-5.6

WatchThis was engineered in Codex, with GPT-5.6 providing the reasoning for the repository-aware development loop. Codex supplied the environment for inspecting and editing the codebase, running checks, and comparing results; GPT-5.6 helped reason across the product, process, and data boundaries. The work remained human-directed, especially for privacy decisions, supported behavior, and release scope.

Concrete contributions included:

• Architecture and contracts: decomposing the product into the React renderer, narrow preload IPC bridge, Electron main process, .NET capture sidecar, optional browser bridge, provider adapters, and Skill compiler; then keeping their versioned TypeScript and Zod contracts consistent.
• Cross-process debugging: following recording and analysis state across renderer, IPC, Electron, the JSON-lines native helper, browser sessions, and model requests to isolate cancellation races, late responses, missing telemetry, and endpoint-protocol mismatches.
• Regression coverage: turning invariants and discovered failure modes into Vitest coverage and executable GUI smoke checks for 60-second segmentation, trace alignment, cancellation, settings, provider compatibility, Skill packaging/import safety, and packaged capture.
• Five-language delivery: maintaining matching application and browser-extension copy across English, Simplified Chinese, Japanese, Korean, and French, backed by locale-parity checks.
• Windows packaging: assembling the Electron build, verified icon assets, self-contained .NET 9 capture helper, MIT license resources, and assisted NSIS installer, with packaged-app smoke-test hooks for the riskiest paths.

Representative evidence is available in the IPC contract, trace schema, analysis pipeline, capture integration, application locales, extension locales, GUI smoke scripts, and packaging configuration.

This is development provenance, not a claim that every WatchThis analysis runs on GPT-5.6. At runtime, the user explicitly chooses and configures one of the supported model transports described below.

Judge walkthrough

The fastest full-capture path is on Windows with Node.js, npm, and the .NET 9 SDK:

npm install
npm run native:publish
npm run dev

1. In Settings → Model, configure and test a vision-capable provider. End-to-end analysis requires an API key or compatible endpoint.
2. Record a short, non-sensitive workflow in WatchThis, or upload a video you are authorized to analyze.
3. Run the analysis and inspect the inferred goal, steps, format, evidence gaps, and clarification questions in Review.
4. Confirm the workflow, then inspect the saved Skill package or export it as a ZIP. Install into Codex only on a machine with interactive Computer Use support.

For a credential-free repository check, run:

npm run typecheck
npm test
npm run build

Without the native build, the app can still start but the Windows interaction collector is reported as unavailable. Packaging and targeted smoke commands are listed under Validate and package.

Evidence captured during a live recording

The Windows native capture helper can supplement the video with:

• mouse position, clicks, wheel input, and drag events;
• key events and hotkeys, with ordinary printable-key identity redacted by default;
• foreground window, process, window title, and focus changes;
• UI Automation / accessibility element context;
• clipboard format, length, and bounded digest metadata, with clipboard text disabled by default;
• watched-directory create, change, rename, and delete notifications; and
• optional browser URL and semantic control events from the bundled Chrome/Edge extension.

File-system notifications do not reliably prove that a file was opened or that a change came from clicking Save. An uploaded video also cannot reconstruct telemetry that was not captured at recording time. WatchThis treats these cases as evidence limits instead of inventing missing events.

Model connections

Configure the provider in Settings → Model. WatchThis currently supports three transports:

Provider	Media transport	Endpoint expectation
Google Gemini	Uploads the source video through the Google Files API, then analyzes timestamped logical windows	Official Gemini Developer API
Gemini-compatible / New API	Sends locally extracted, timestamped image frames as Gemini inlineData; it does not require the Google Files API	Native Gemini generateContent protocol
OpenAI-compatible	Sends timestamped image frames and the aligned trace through a Chat Completions-style request	A vision-capable OpenAI-compatible Chat Completions endpoint

Compatibility labels describe the request protocol, not the capabilities of every proxy or model. Use Test and verify that the selected endpoint supports image input and sufficiently long structured output.

Skill packages and agent compatibility

WatchThis exports a validated ZIP containing a human-readable SKILL.md, Codex metadata, a WatchThis manifest, and the platform-neutral workflow specification. The Skill library preserves generated ZIPs for re-export, Codex installation or update, and removal. Existing compatible ZIP packages can also be imported into the library.

The target agent platform must be able to:

• observe the current screen;
• control the pointer and keyboard; and
• keep the user in control of confirmations and consequential actions.

A platform that can read the ZIP but cannot perform interactive Computer Use cannot execute the skill. Treat the first run of every generated skill as a supervised validation, and verify the final result rather than assuming the recorded demonstration generalizes perfectly.

Privacy and sensitive data

Screen recordings can contain passwords, personal documents, financial data, messages, notifications, and other confidential information. Window titles, accessible names, relative file paths, URLs, and generated examples may also be sensitive even when body text is redacted.

Before recording or analyzing:

• record only computers, accounts, and data you are authorized to use;
• close unrelated applications and notifications;
• avoid displaying secrets, API keys, payment details, health data, or private messages;
• review the selected model provider and its retention policy; and
• inspect the generated workflow before exporting or sharing it.

Saved provider credentials are managed by the Electron main process and use the operating system's safeStorage encryption when available. If safeStorage is unavailable, the current implementation stores the API key in plaintext in provider-settings.json and requests user-only file permissions; protect your OS account and do not share this file. Official Gemini analysis uploads the video to Google; compatible transports send sampled screen frames to the Base URL selected by the user. A sampled frame can still contain everything visible on screen.

WatchThis recordings are stored locally in the directory shown under Settings → Storage. Changing that directory affects future recordings and does not move or delete existing files. The application does not currently apply automatic retention to recordings, so remove them manually when they are no longer needed. Deleting a Skill library entry does not delete exported or installed copies.

Read docs/privacy.md for the current data boundaries and docs/architecture.md for evidence and pipeline details.

Requirements

For development and packaging:

• Windows is recommended and required for the complete native capture path;
• Node.js with npm; and
• the .NET 9 SDK for building the self-contained Windows capture helper.

End users of the packaged x64 build do not need a separate .NET installation. An API key or a compatible endpoint is required for real model analysis. The optional browser telemetry path requires Chrome or Edge.

Development

Use the commands in the Judge walkthrough to start the development build. Optional developer environment variables are documented in .env.example; the in-app settings are the normal configuration path.

To load optional browser telemetry during development:

1. Open chrome://extensions or edge://extensions.
2. Enable developer mode.
3. Load browser-extension as an unpacked extension.
4. Start a WatchThis recording and connect the one-session endpoint and token shown by the application.

The extension removes URL fragments, credentials, and query values, and sends limited semantic control information rather than the complete DOM. It only records after an explicit session connection.

Validate and package

npm run typecheck
npm test
npm run build
npm run native:publish
npm run package:dir
npm run package:win

npm run package:win renders and verifies the application icon, publishes the native helper, builds the Electron application, and creates release/WatchThis-<version>-x64.exe. The assisted Windows installer displays the MIT license and includes a copy in the installed resources.

GUI smoke tests are available under the smoke:* npm scripts. Run the relevant smoke tests after changes to recording, cancellation, settings, provider compatibility, or packaging.

Repository layout

Path	Purpose
src/main	Electron main process, capture lifecycle, providers, analysis, settings, and Skill storage
src/preload	Narrow IPC bridge exposed to the renderer
src/renderer	React user interface and five-language localization
src/shared	Versioned contracts shared across processes
native/WatchThis.Capture	Windows native interaction capture helper
browser-extension	Optional Chrome/Edge telemetry bridge
docs	Architecture and privacy details
scripts	Icon, packaging, endpoint, and GUI smoke checks

Contributing

Issues and pull requests are welcome. Keep changes focused, describe the user-visible behavior, and include tests for logic that can regress. Before submitting a change, run the type check, unit tests, and the relevant build or smoke tests.

Do not commit API keys, provider settings, real recordings, screenshots containing private data, .watchtrace files, or generated artifacts containing user data. Use synthetic fixtures and redact logs before sharing them.

License

WatchThis is released under the MIT License.

Copyright (c) 2026 AlanBacker.