feat(media): transparent authoring proxies for browser-hostile codecs#2462
Open
miguel-heygen wants to merge 3 commits into
Open
feat(media): transparent authoring proxies for browser-hostile codecs#2462miguel-heygen wants to merge 3 commits into
miguel-heygen wants to merge 3 commits into
Conversation
Codec manifest (studio-server) keyed by root-relative URL pathnames, on-demand H.264 proxy transcoder with per-key dedupe + global bound and a project-local .transcode-cache, proxy negotiation (?hf-proxy=h264) on the studio preview route, hyperframes play (which also gains byte-range serving), and the static project server behind check/snapshot/compare; runtime capability check with proactive swap, zero-videoWidth reactive rescue, WeakMap state eviction, and a structured diagnostic; check pre-resolves hostile proxies before its timed render-ready phase; media.autoProxy config + --no-proxy flags; asset-resolution helpers unified into @hyperframes/parsers. Render paths never see proxies.
…sh time Splits createPublishArchive into exported build/zip halves (thin composition preserved, cloud render byte-identical with a regression test), generalizes rewriteHtmlAttributes, and bakes _proxy/ entries with <video src> rewrites for hostile assets behind --proxy/--no-proxy and media.autoProxy. Audio elements stay on originals (AAC demuxes from HEVC containers). Also: check surfaces the runtime proxy-fallback console line as an info finding (stable diagnostic token embedded in the message), lint's hevc_preview_codec message now describes auto-proxying, docs and skills updated to match.
Review fixes across the proxy stack: proxyTranscoder committed as UTF-8 text (literal NUL delimiters became escapes; the file was binary to git), ffmpeg children get a hard timeout so a hang can never wedge the global transcode slots, failed transcodes are negative-cached so pre-warm stops respawning doomed encodes, and client aborts after a held response no longer write to a dead socket. Alpha sources (ProRes 4444 etc.) are never proxied anywhere, preview/Studio now honor media.autoProxy and --no-proxy, the runtime swaps only when a codec map is present (opt-out surfaces stay inert) and skips browser-safe entries, the unavailable diagnostic is visible to check and devtools, play serves proxies as video/mp4 with mov/m4v/mkv recognized, the publish bake resolves srcs exactly like the scan (percent-encoded srcs now rewritten), codec probing is genuinely async off the event loop, and codec-map injection is one shared implementation across surfaces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Transparent authoring proxies: any video codec the project's FFmpeg can decode now plays on every live surface. When the viewing browser cannot decode an asset (HEVC being the motivating case), the preview/studio/play servers transcode a cached H.264 proxy on demand (
?hf-proxy=h264, response held until ready,.transcode-cache/keyed by path+mtime+size), the runtime swaps the<video>element's source with a visible diagnostic,checkpre-resolves proxies before its timed phase and surfaces the diagnostic as an info finding, andhyperframes publishbakes proxies into the archive so static player pages work in HEVC-less browsers. Render is untouched by construction and by regression test: it always uses originals.Opt out per surface with
--no-proxyor per project withmedia.autoProxy: falseinhyperframes.json.Why
Follow-up to #2453: that PR documented that HEVC assets render fine but preview in a browser without HEVC support shows a silent black frame. This PR removes the failure instead of describing it, and does so for every browser-hostile codec by one mechanism (FFmpeg's codec universe becomes the input contract). Design decisions and verified probe evidence are in the plan (docs/plans/2026-07-14-002, local): notably, an HEVC+AAC file fires no error event in an HEVC-less browser (the AAC track satisfies the demuxer), so the reactive trigger is
loadedmetadatawithvideoWidth === 0, not error events; and<audio>elements never need proxying (audio decode is independent of the video codec).How
@hyperframes/studio-server: codec manifest (media-codec-map, ffprobe-backed, async pool, mtime-cached, keys are root-relative URL pathnames), proxy transcoder (proxy-transcoder, crf 18 veryfast faststart, per-key in-flight dedupe + global 2-slot bound + 15-min kill ceiling + negative failure cache, temp+rename writes), preview-route negotiation with ETag salting and bounded pre-warm at codec-map injection.@hyperframes/coreruntime:mediaProxy.ts— proactive swap at media registration (canPlayTypeagainst coarse representative codec strings), reactive zero-videoWidthrescue, tertiary error-event rescue; render-mode short-circuit; swaps evict the per-element drift/volume WeakMap state; no codec map injected means no swaps (opt-out surfaces stay inert); alpha-bearing sources (ProRes 4444) are never proxied, preserving transparency.playgains byte-range serving and the proxy branch, the static project server behindcheck/snapshot/compare/etc. gains injection + the proxy branch,checkpre-resolves hostile proxies before its 3s render-ready window and reports proxy fallbacks as info findings, andpublishbakes via a new build/zip seam socloud render's archive stays byte-identical (regression-tested).hevc_preview_codecmessage and the docs/skills now describe auto-proxying.Test plan
hyperframes playserver on a cold cache — the video element proactively swaps to the proxy URL, real frames paint (canvas pixel sampling), the paired<audio>element plays the original AAC untouched, exactly one diagnostic line appears, and.transcode-cache/holds an H.264 proxy.--no-proxyverified: no map injection, proxy param 404s.cloud render's archive carries original bytes and unmodified HTML.Known residuals (accepted, documented):
.transcode-cache/has no eviction in v1 (safe to delete; mtime/size-keyed so never stale-wrong); a publish-time TOCTOU exists if an asset is re-exported mid-bake (warn-and-keep-original planned as follow-up);previewin dev-mode/local-studio spawn paths honors the config file but not a forwarded--proxyflag; AV1 assets are baked at publish even though most browsers decode AV1; the studio origin's connection budget under many concurrently-held cold transcodes is mitigated by pre-warm + the 2-slot bound, not eliminated.Post-Deploy Monitoring & Validation
No additional operational monitoring required: local-tooling feature (dev servers, CLI, publish-time transform); no service or render-path impact. The
media_proxy_fallback/media_proxy_unavailablecheck findings are the observable signal if users report proxy issues.