AI Walkthrough Studio

Turn a real screen recording into an edited, captioned product walkthrough. An LDS Project, contributed by Prateek Gaurav.

Build and run this app on your own computer. Paid voice generation is disabled by default. Bring your own recording; no sample footage or API key is included.

Start here

Download the starter ZIP from Releases to follow the build-along exercises. Choose the reference ZIP for the completed app. Both include the guide and recovery checkpoints. Cloning this repository also gives you the completed reference.

You need Node.js 22.13+ and FFmpeg/FFprobe on your PATH. On macOS, FFmpeg is available from Homebrew with brew install ffmpeg. No Remotion or cloud rendering subscription is required. Extract the ZIP, open a terminal inside its ai-walkthrough-studio folder, then run:

npm ci
npm run setup
npm start

Open http://localhost:3076. The server binds only to this computer. Stop it with Ctrl+C. Start it again with npm start; your projects and exports persist.

Bring your own recording: choose New recording and select an MP4, MOV or WebM. A 20–40 second clip is a comfortable first build. Record one simple workflow in a product you know, with private information and notifications hidden. No practice video is supplied or downloaded. Original audio is removed on import.

Make your first demo

1. Import a recording. Choose a clip up to three minutes and 250 MB. The original stays intact. A silent 720p editing copy and filmstrip are prepared locally.
2. Analyze with Gemini. This is optional. Confirm the recording you want to send. Only a silent, smaller analysis copy goes to Gemini. One response is cached; restoring the draft uses that cache. You can also write every scene yourself. The default is gemini-3.8-flash with low thinking and explicit 2 fps sampling. Set GEMINI_MODEL in your private env file if your account uses another model. New accounts should use a current model; Google restricts Gemini 2.5 to existing users.
3. Watch each scene. Click its timeline segment. Check the visible evidence and edit the title, in/out points, and voiceover. Split, remove, or reorder scenes as needed. Removing a scene does not delete any original footage.
4. Frame the moment. Use zoom and horizontal/vertical focus when small controls need attention. The same crop is used in the exported MP4. Captions are optional.
5. Save and export a silent draft. Rendering happens locally. Under Saved exports, use MP4, Captions and Script. Script is the frozen scene plan for that export, not your later edits. Silent-draft caption timing is estimated, not a speech alignment. Watch the whole export before spending voice credits.
6. Review the script. Check every line against the footage. Shorten any line that cannot be comfortably spoken within its scene. Mark the script reviewed and approve it. An edit invalidates that approval.
7. Voice is the final step. Keep paid generation locked during development. After final review, use the procedure below. Listen to every generated scene before exporting the narrated version.

Protect your ElevenLabs allowance

Nothing calls ElevenLabs on import, load, edit, save, analysis, preview, or export. Even a direct API call to the narration endpoint is rejected while locked.

When the recording, script and timings are final:

1. Set ELEVENLABS_API_KEY and ELEVENLABS_VOICE_ID (and a display name in ELEVENLABS_VOICE_NAME) in the private env file. Use a voice you own or are allowed to use.
2. Choose a conservative TTS_MAX_CHARACTERS ceiling (default: 1,600 per request). This is a character budget. Reading-time estimates are not a provider billing quote or an exact prediction of generated duration.
3. Set ENABLE_PAID_NARRATION=true and restart. This only unlocks the final action; it does not generate anything.
4. Approve the current saved script. Select Review voice generation, inspect the new characters and cached lines, and explicitly confirm.
5. Each successfully generated line is saved immediately, along with its actual word alignment. Unchanged text with the same voice/settings is reused. Fixing a crop or timing does not regenerate audio; changing a line regenerates that line.
6. If audio outlasts a scene, generation stops before spending on further lines. Extend that scene using the saved audio. Exports never silently cut speech off.
7. Failed or interrupted provider requests are never automatically retried. An unresolved usage receipt blocks further generation. Check ElevenLabs history; recover saved audio first. See docs/RECOVERY.md.

While building and testing, keep ENABLE_PAID_NARRATION=false. No paid speech is needed to test the entire editing/export flow.

What stays local, what leaves

• .data/ contains recordings, edits, thumbnails, cached analysis, audio and exports.
• .env.local holds credentials; it is gitignored, never bundled, and not served.
• Clicking Analyze this recording sends the silent proxy to Google's Gemini API. Use a recording you are comfortable sending to that service; avoid visible secrets.
• Explicit final narration sends only approved script lines to ElevenLabs.
• Rendering uses your CPU through FFmpeg. No video rendering/upload service is used.
• This app has no analytics, third-party browser scripts, remote fonts or auto-publish.
• A loopback server is for local use, not an authenticated multi-user production service. Do not expose its port through a tunnel or deploy it as-is.

How to build it yourself

Read the guided build for six practical steps. The starter leaves three functions for you to implement in learner/; the reference completes them. Run npm run check:storyboard, npm run check:timeline, and npm run check:captions as you go. Recover an individual file with npm run restore -- timeline; your current file is backed up first. The interface, provider contracts, storage and renderer are separate modules so the project is understandable and extendable.

npm test       # Includes a real FFmpeg import/render smoke test. No paid API calls.
npm run build # Bundles the browser app only; no env substitution.

Automated checks

The GitHub workflow runs the offline suite on Linux, macOS and Windows, including a real FFmpeg import and render. It has no provider credentials and cannot generate paid speech. A green run verifies these automated checks; review your own exported video as well.

Current boundaries

• One landscape output template: H.264 MP4, 1280 × 720, 24 fps.
• Maximum 24 scenes and three minutes of edited footage.
• Static per-scene focus crops; no automatic camera moves or invented interactions.
• Source audio is deliberately muted to prevent old narration fighting the new script. A draft is silent until approved generated voice is included.
• Gemini works from a 2 fps, 960 × 540 proxy. Very fast actions/small labels require human checking in the full editing preview. Its output is a draft.
• Edits are explicitly saved. A close-tab warning protects unsaved changes. Revision checks prevent one tab overwriting newer edits from another.
• Voice integration has offline coverage plus a real local narrated reference export. Your copy ships without a key or voice ID and keeps generation disabled.

References

• Gemini video understanding
• Gemini generateContent API
• ElevenLabs speech with timing
• FFmpeg filters

License

Original LDS code is licensed under Apache 2.0. See LICENSING.md for third-party terms, earlier licenses and the scope of this grant. Maintained by Let's Data Science.