fix(windows): PTT hold timing, error surfacing, mic permission + verified setup wizard - #10
Merged
Merged
Conversation
…ion tuning
Walkthroughs
- New [POINT:...] semantics: a sequence of tags becomes a numbered
walkthrough rather than a single label. Cursor walks step-by-step,
each step gets a pulsing halo and a numbered bubble (e.g. "2/4
click File"). Single-point answers fall through as 1-step
walkthroughs and look the same as before plus the halo.
- Stream window renders a walkthrough card in lockstep with the
overlay: the active step is highlighted, completed steps get a
strike-through and check, pending steps are dimmed.
- Step timing is driven from the main process (companion-manager)
and broadcast to overlay + stream via WALKTHROUGH / WALKTHROUGH_STEP
IPC, so they can never drift. Per-step dwell scales with caption
length (2.6–5.5s) so longer instructions stay readable.
- Stream window stays visible across a walkthrough in 'responses'
mode, instead of hiding the moment voice state goes back to idle.
Screenshot quality
- captureDisplays now defaults to cursor-only — only the screen the
cursor is on is sent to the LLM. Less wasted vision tokens, no
ambiguity over which screen the model is talking about.
- Empty thumbnails (the macOS "screen recording denied" pattern) are
filtered out instead of being uploaded as empty base64. If all
thumbnails come back empty, the user gets a friendly spoken
response telling them to grant Screen Recording, instead of an
Anthropic 400.
- Bumped MAX_DIMENSION from 1280 → 1600 and quality 80 → 82 so the
model has more pixel precision to land [POINT:...] coordinates
on small UI targets.
Push-to-talk modes
- New `pttMode: 'hold' | 'toggle'` setting, exposed as a segmented
control in General → Shortcut. Hold = press-and-hold (Win/Linux);
Toggle = tap to start, tap to stop. macOS forces toggle regardless
of stored value because Electron's globalShortcut exposes no
key-up event there.
Clipboard typing
- New [TYPE:text] tag the model can emit when the user asks to
type something into a focused field. Text is copied to the
clipboard and a glassy toast appears over the cursor display
("Copied — press ⌘V to paste") with a preview of the text.
Stripped from the spoken/streamed reply so TTS doesn't read it
back. Setting toggle ("Allow Flicky to type for you") is in
place; native auto-typer wires up next commit.
Transcription tuning
- Groq Whisper now gets `language=en`, `temperature=0`, and a vocab
prompt loaded with the kind of words a screen-aware assistant
hears (UI verbs, common app names). Cuts a lot of homophone
weirdness and language-detection drift on short clips.
TTS echo fix
- TTS audio was being broadcast to every overlay window, so users
with N displays heard N copies. Added sendToOneOverlay so audio
plays exactly once.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the "Allow Flicky to type for you" setting is on, [TYPE:...]
tags are now sent through @nut-tree-fork/nut-js, which fires real
global keystrokes via libnut. The text shows up in whatever field
the OS considers focused — no paste step.
Graceful fallback: if the native module fails to load, the
Accessibility permission isn't granted, or the type call throws,
we silently fall back to clipboard handoff. The user is never
left with no way to act on the request.
Permissions:
- Accessibility status is now part of getPermissions() on darwin.
- Banner gets a third row ("Accessibility") that only shows when
autoTypeEnabled is on AND the perm is missing — keeps the banner
quiet for users who don't use the feature.
- Flipping autoTypeEnabled on triggers the macOS Accessibility
prompt right then; Grant button on the banner deeplinks to the
Privacy_Accessibility pane and re-prompts in case the dialog
was dismissed.
UX:
- Toast distinguishes auto-typed (⌨️ "Typed for you") from
clipboard (📋 "Copied — press ⌘V to paste").
- General-tab copy updated to reflect that the feature is real,
not "coming soon".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several issues that compounded on multi-display setups, plus the bug
behind the "weird transcription" the user noticed earlier.
Audio capture (correctness)
- Mic START_CAPTURE / STOP_CAPTURE were broadcast to every overlay,
so each monitor opened its own getUserMedia + AudioContext and
fanned chunks back to main. companion-manager appended every
chunk to the same buffer, so Whisper was receiving N interleaved
copies of the audio. Routed through sendToOneOverlay so exactly
one overlay captures.
Cursor IPC fanout
- 60fps poll broadcast to every overlay even when "Show cursor"
was disabled. Now: skips the entire poll when disabled, runs at
~30fps, and sends position only to the overlay whose display
contains the cursor. When the cursor leaves a display, that
overlay gets one `{ off: true }` pulse to clear its
isCursorOnThisDisplay state.
Settings store
- get / getAll / set re-read the JSON file from disk on every call.
emitSettings() pulls getSettings() repeatedly, so a single PTT
release was doing several disk reads. Now hydrates once into
an in-memory cache; writes still persist to disk.
Stream window lifecycle
- Created eagerly at boot regardless of `streamVisibility`. Default
is 'off', meaning a fresh-install user had a Chromium renderer
running in the background to receive IPC nobody would ever see.
Now lazy-created on first non-'off' visibility, and *destroyed*
(not just hidden) when set back to 'off'.
Permissions polling
- 1.5s polling interval ran forever, even when the panel was hidden
in the tray. Polling is now bound to panel show / hide events and
bumped to 5s — the banner only renders in the panel anyway.
Walkthrough IPC
- WALKTHROUGH and WALKTHROUGH_STEP fanned to every overlay even
though only the overlay containing the step's coordinates renders
it. Main now resolves the target overlay once on walkthrough
start (steps[0]) and sends the events only to that one. Stream
still receives both since its card always renders.
Display-info race
- One-shot push on did-finish-load could fire before the renderer's
IPC listener attached, leaving overlays with no display bounds
forever — the fallback then defaulted isCursorOnThisDisplay to
true, so the companion cursor appeared on every monitor. Added
a `get-display-info` invoke handler keyed by sender webContents
so renderers can pull bounds on mount, and flipped the bounds-
unknown fallback to hide instead of show.
Walkthrough rendering
- showOnThisDisplay didn't gate on the step's display during
navigating/holding modes, so the annotated cursor + bubble
appeared on every monitor at once. Now gated on
isStepOnThisDisplay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ScriptProcessorNode is deprecated for cause: it runs the audio
callback on the renderer's main JS thread, and is the documented
source of crackle / dropouts under load. Replaced with an
AudioWorkletNode whose processor lives on the dedicated audio
rendering thread.
Mic stack lifecycle
- Previously the AudioContext, MediaStream, and ScriptProcessor
were torn down on every PTT release and rebuilt on every press.
getUserMedia resolution is 100–300ms on macOS — that latency was
paid every turn.
- Now: built lazily on first PTT and kept warm across turns.
Start / stop just toggles a flag inside the worklet via
port.postMessage, which is essentially free. Real teardown
(release tracks, close ctx) only on overlay unmount.
- Worklet posts Int16 PCM as transferable ArrayBuffer so chunks
are zero-copy from audio thread → renderer → main IPC.
Vite worklet bundling
- Worklet is plain JS (workers/worklets must be) at
src/renderer/audio-capture-worklet.js. Loaded via
`new URL('./audio-capture-worklet.js', import.meta.url).href`,
which Vite inlines as a `data:text/javascript;base64,...` URL
in production — no separate asset to ship, no file:// path
resolution quirks in the packaged Electron app.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stream auto-scroll - The body element's scrollTop was rewritten on every chunk arrival, forcing synchronous layout token-by-token, and yanked the user back to the bottom even if they had scrolled up to read an earlier turn. Now: only auto-scrolls when the user is already pinned within ~24px of the bottom, and defers the scroll write to the next animation frame so a fast token stream doesn't thrash layout. Vite manualChunks - Three independent renderer entries (panel/overlay/stream) each inlined react + react-dom + jsx-runtime, so the installer shipped React three times. Pulled shared deps into one `react-*.js` chunk the engine caches across windows. Runtime platform via preload - `vite.config.ts` previously baked `process.platform` at build time, which would leak the build host's platform into the renderer when cross-compiling (CI on Linux building a macOS dmg shipped 'linux' to the renderer). Removed the compile-time inline; preload now exposes `window.flicky.platform` resolved at runtime in main. All renderer call sites updated. Skipped: "destroy overlays when cursor disabled" - The audit recommended destroying overlay BrowserWindows when the user toggles "Show cursor" off. On closer reading this would regress unrelated features — overlays also host TTS playback, mic capture, walkthrough rendering, and the type-fulfilled toast. Group 1 already cut idle overlay cost to near-zero (cursor poll skipped when disabled, single-target IPC for mic/TTS/walkthrough/ toast), so leaving them alive at low cost is the right tradeoff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sandbox renderers - All three windows (panel, overlay, stream) now run with sandbox: true. The preload only uses contextBridge + ipcRenderer — both work in sandboxed mode — so this just closes off Node API surface to the renderer with no behavioral change. Walkthrough lifecycle race - walkthroughActive flipped false only when onWalkthrough(null) fired, but the scheduler emits onWalkthroughStep(null) first. In the tiny window between those two events a status reader could observe walkthroughActive=true with no current step. Now cleared from both signals. Display topology diff - display-added / display-removed used to call rebuildOverlays(), which destroyed every overlay window and recreated them all. syncOverlaysToDisplays() now diffs against the current set: destroys overlays whose display is gone, creates overlays for newly-attached displays, and leaves the rest untouched. PTT toggle resync - If startPushToTalk silently failed (e.g. transcription provider init threw), companion's isRecording flipped back to false but the local pttActive stayed true. Next tap would issue a stop on nothing. Exposed CompanionManager#recording and the toggle handler now reconciles pttActive after each call resolves. PostHog lazy import - posthog-node and its transitive deps were imported at boot even though analytics are off by default (apiKey is empty). Lazy-import inside initAnalytics so the module only hits the JIT when an actual key is supplied. Saves cold-start latency in packaged builds. Removed unused `ws` dep - grep across src/, scripts/, and landing/ found no references. Dropping ws + @types/ws. clearStream IPC self-loop - ipcMain.on(IPC.CLEAR_STREAM, () => sendToStream(IPC.CLEAR_STREAM)) was a renderer→main→same-renderer round trip just to clear local React state. Stream's "clear" button now updates its own state directly. Removed the dead ipcMain handler, dead preload exports (clearStream, onClearStream), and the now-unused channel listeners. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Group 4's sandbox flip broke the renderer — all three windows came up as blank black screens. Likely cause is the relative preload require-resolution path under sandboxed mode; the preload's own imports (../shared/types via tsc-emitted CommonJS require) don't play nicely until the preload is bundled as a single self-contained file (e.g. via esbuild). Reverted sandbox to false on panel/overlay/stream. Other Group 4 changes (walkthrough cleanup, display diff, ptt resync, posthog lazy-import, ws removal, clearStream cleanup) are unaffected and remain in place. If we want sandbox later, the prerequisite is a build step that bundles preload/index.ts into a single .js with its dependencies inlined. Out of scope for this audit pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The overlay's `closed` listener used `win.webContents.id` to clean up the overlayDisplayByWebContents map, but by the time `closed` fires the window's webContents has already been destroyed and property access on it throws "Object has been destroyed". Crashed on app quit. Capture the id at construction time and close over the primitive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Group 1's cursor-only screenshot was a real win for token cost, but the filter is brittle — if `screen.getCursorScreenPoint()` returns coords that don't fall inside any reported display bounds (mid resolution change, post-wake, or topology states we don't recognize), the filter excludes every display and we fall through to the "no captures" error path, telling the user we can't see their screen even when permissions are fine. Now: if cursor-only excludes everything, log the cursor + bounds to the console (so we can diagnose if this happens again) and fall back to all displays. The model still gets a screenshot to work with; it just isn't pre-narrowed to one screen for that turn. Also: the "can't see your screen" path used to update the chat text but skip TTS, so users with their attention elsewhere got silence and no audible signal anything went wrong. Now the error message goes through TTS too when speakReplies + ElevenLabs are configured. And added a console.warn when captureDisplays returns zero screenshots, with display + source ids, so future regressions in this path are diagnosable from one log line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two real bugs from the audit-fix loop, found via the user's
diagnostic log:
[Flicky] captureDisplays produced zero screenshots.
displays=1, sources=1, sourceIds=[1], displayIds=[1]
Screen capture (group 1)
- Added `thumbnail.isEmpty()` and `size === 0` guards in screen-
capture.ts to defend against the macOS "permission denied →
empty thumbnail" failure mode I'd theorized. The user's log
confirms the source matched correctly and *something* came
back, but those guards still rejected the thumbnail. The
hypothesis was wrong, the guard was overzealous, and it broke
capture on at least one valid-permission configuration.
- Removed the over-broad guards. Replaced with a narrower one
that only bails when `toJPEG()` actually returns 0 bytes —
the precise condition that triggers Anthropic's
"image cannot be empty" 400 — and logs the thumbnail size,
target size, and scale factor so a future regression in this
path is diagnosable from one log line.
PTT after first turn (group 2)
- The warm-mic-stack rewrite reset `micStopRequestedRef` only
inside `ensureGraph`. On press 1, the graph is built and the
flag is reset. On press 1's release, stopMic sets the flag
to true. On press 2, ensureGraph short-circuits with the
existing graph and never resets the flag, so startMic's
guard (`if (... || micStopRequestedRef.current) return`)
bails and the worklet never gets the 'start' message. Mic
stayed muted from press 2 onward.
- Reset the flag at the top of startMic so every press gets
a fresh start, while preserving the start/stop race
guarantees (a stop landing during the await still wins
because it sets the flag back to true after we cleared it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On macOS, desktopCapturer's source thumbnails can come back empty on the very first call shortly after app launch — the capture pipeline hasn't warmed up yet, so the first PTT after relaunch shipped no screenshot to the model. A single 300ms-delayed retry reliably hands back populated thumbnails on the second attempt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fied setup wizard Windows fixes: - hold-to-talk stopped after 250ms because the OS key-repeat delay (500ms-1s) hadn't elapsed yet; first fire now gets a 1.1s grace window - transcription/model/mic failures were console-only; now surfaced in the panel + stream, and a failed transcription no longer leaves voice state stuck on 'listening' - query Windows microphone privacy status and deep-link to ms-settings - second-instance launch shows the panel instead of silently exiting - tray click can hide the panel again; overlays follow display-metrics changes; .ico tray icon; hide stock menu bar - shortcut capture uses e.code so Shift combos / non-US layouts bind - OpenAI whisper provider used the Anthropic key; version label hardcoded Setup wizard (first run, re-runnable from General): permissions, provider keys validated by a real round-trip before saving, shortcut press check, live mic level check, and an end-to-end trial turn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Resolves conflicts with #6 (safeStorage fallback), #7 (display-info via launch args) and #8 (GPU crash guard). Keeps master's synchronous display seeding and drops the branch's async get-display-info IPC; keeps the branch's per-webContents display map (used for cursor routing, walkthrough targeting and bounds resync) and the error strip alongside master's unencrypted-keys banner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XSbPaf2fxjrLNjNnn68bXW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
listening.getUserMediafailures..icotray icon; stock menu bar hidden; shortcut capture usese.code.Test plan
npm run typecheck/npm run buildv1.2.1after merge → GH Actions builds and publishes the Windows.exe🤖 Generated with Claude Code