feat(mediaplayer): VP9 decode via OS codecs + WebM demuxer with Cues seek#958
Draft
towneh wants to merge 11 commits into
Draft
feat(mediaplayer): VP9 decode via OS codecs + WebM demuxer with Cues seek#958towneh wants to merge 11 commits into
towneh wants to merge 11 commits into
Conversation
…decs An MP4 whose only video track carries a codec the demuxer doesn't recognise used to have that track silently dropped in parse_trak, leaving audio-only playback under a black screen. Track the mdia/hdlr handler type and the first unrecognised stsd entry fourcc; after the moov parse, a 'vide'-handler track with no supported codec and no supported video track alongside raises "video codec 'xxxx' is not supported" naming the fourcc. Files with a supported video track next to an unsupported one keep playing the supported track, and audio-only files are unaffected. Track selection is by role while at it — the first supported video track and the first supported audio track, wherever they sit in the moov. The previous first-two-traks cut would leave a valid file ordered audio,audio,video silently video-less, bypassing the new error entirely.
Adds BASIS_CODEC_VP9 to the sink codec enum (AV1 = 4 is reserved) and teaches the MP4 demuxer the vp09 sample entry: width/height come from the VisualSampleEntry fixed fields (VP9 carries no out-of-band decoder config, so vpcC is skipped and extradata stays empty), and both sample walks emit VP9 samples raw — no Annex-B rewrite, superframes fed whole, per the AU contract note in basis_media_internal.h. Keyframes use stss when present, else a new basis_vp9_is_keyframe helper that reads the uncompressed-header head bits (superframe-safe: the index trails the buffer, so offset 0 is always the first sub-frame's header). The CEA-608 caption scan is now gated to H.26x formats: the caption walker is an Annex-B NAL walk, and raw VP9 samples can contain 00 00 01 runs it would misparse into the 608 decoder. Decode backends reject the format until their VP9 mapping lands in the follow-up commits, so this is demux-layer plumbing only.
New protocol/basis_webm.{c,h}: a narrow WebM/Matroska demuxer feeding
VP9 into the sink — the container YouTube's >1080p VP9 ladder ships in
(video-only WebM legs; audio stays on the m4a split leg). Dispatch
sniffs the EBML magic (1A 45 DF A3) ahead of the TS default, with a
.webm extension fallback; googlevideo URLs are extensionless so the
sniff is what matters.
Walk shape: EBML header limits validated (MaxIDLength <= 4,
MaxSizeLength <= 8, DocType webm/matroska); Segment children accepted
in any order; only Segment/Cluster may be unknown-size (an unknown-size
Cluster ends at the next Segment-child ID, which the ID-driven dispatch
gives for free); clusters stream child-by-child — only individual block
payloads are buffered (64 MiB cap; whole-buffered header elements get a
much tighter 16 MiB, real ones being KBs). Block-relative timestamps
are signed s16, and TimestampScale is remote content — bounded to 1
s/tick — so all timestamp math saturates in int64. All three lacing
schemes parse defensively. The CodecID switch announces the first V_VP9
video track; any other video CodecID raises the unsupported-codec
error; audio/subtitle tracks are skipped (the Opus and AV1 items extend
the same switch).
Seek rides the Cues index at cluster granularity: Cues ahead of the
clusters parse inline; a trailing Cues (default ffmpeg mux) is
ranged-fetched at open via the start-of-Segment SeekHead when the
source can reposition. Cue times are stored in ticks and converted at
seek time (Segment children arrive in any order, so the scale may not
be final when Cues parse), and each CuePoint's positions are filtered
to the selected video track's CueTrack, first position as fallback.
Duration is reported only when the index and a repositionable source
are both in hand, so a reported duration always means the seek bar
works — cueless or forward-only streams keep duration 0.
Verified closed-loop against the mr.town codec fixtures (cues-front /
trailing-cues / no-cues / VP8-negative): full-file AU counts match the
encoder's packet count, trailing-cues fetch reports the same duration
as inline, forward/backward/near-end seeks land on cue'd keyframes at
or before target, and the cueless file plays forward-only with no
duration and seeks ignored.
video_subtype maps VP9 to MFVideoFormat_VP90 (the Store 'VP9 Video Extensions' MFT, or a vendor one); the MFT-missing error names the Store package. VP9 carries no out-of-band decoder config, so the input type leans on the frame size both demux paths always announce — there are no parameter sets for the MFT to parse in-band. Two guards close the silent-black-screen hole a software-falling-back MFT would open (the Store extension decodes on CPU when the GPU lacks hardware VP9, and those samples arrive without DXGI backing, which the drain can only discard): - the video drain now raises a clear error on the first non-DXGI output sample instead of silently dropping every frame; - a new capability probe, basis_media_probe_video_codec(codec), answers only when BOTH legs pass: a decoder MFT enumerates with exactly create_video_mft's flags, AND the GPU's D3D11 video device lists the codec's decode profile (the check DirectX-accelerated players perform; an MFT-only probe is a false positive on no-hw GPUs). Engine-less, any-thread, cached for the process lifetime — meant for resolver-side format selection. 8-bit profile 0 only for VP9; 10-bit falls to the drain guard. AV1 probes 0 until its decode path lands. The stub backend reports 0 for every codec.
set_video_format maps VP9 to the video/x-vnd.on2.vp9 mime; VP9 carries no decoder config so the csd-0 branch is naturally skipped. The capability probe answers with a cached createDecoderByType check — every Quest hardware-decodes VP9, so the codec this probe exists to gate is universally hardware on the Android target. AV1 probes 0 until its decode path lands. The OS-extractor URL path is mime-driven and needed no change; the portable demuxer covers extractor-rejected URLs.
basis_media_probe_video_codec P/Invoke plus a public BasisNativeVideoSource.IsVideoCodecSupported(BasisVideoCodec) — the surface resolvers use to gate format selection by what this platform actually decodes. Binaries that predate the export answer through an EntryPointNotFoundException guard as they behaved before the probe existed: H.264/H.265 decode, VP9/AV1 don't.
… them BestVideoOnly gains a probe-gated VP9 lane: where IsVideoCodecSupported(VP9) passes (hardware decode confirmed, verdict cached), video-only VP9 rungs in WebM or MP4 carriage are eligible up to 2160p — YouTube's above-1080p ladder is VP9 in video-only WebM, so this unlocks >1080p on effectively all uploads, paired with the existing m4a audio leg over the split path. Profile 0 only: bare 'vp9' or dotted 'vp09.00.*' (yt-dlp signals profile 2/10-bit as the distinct 'vp9.2'), since profile 0 is what the decode probe answers for. SDR only: the renderer has no tone mapping (HDR uploads keep playing via their parallel SDR ladder). Ranking is height first, avc1 over vp9 at equal height, then bitrate — a VP9 rung only wins where it offers more height than every eligible avc1 rung, so on YouTube's usual complete avc1 ladders selection at or below 1080p is unchanged. BestAudioOnly and BestMuxed are untouched (YouTube's muxed WebM is legacy VP8+Vorbis).
Content rows for the mr.town VP9 fixtures (WebM/MP4 lanes, the three Cues placements with the duration-implies-seek invariant, the unsupported-codec negatives, the software-fallback guard), a WebM Cues-seek behaviour checklist, resolver scenarios for probe-gated selection (VP9-capable vs not, HDR, AV1), and the README format/limits text both packages carry.
Both RIST-enabled (-DBASIS_WITH_RIST=ON; librist statics from the media-native CI artifacts). The Windows DLL passed the standalone harness before commit: probe legs (H264/H265/VP9 = 1 on a VP9-capable box, AV1 = 0, invalid ids rejected), WebM and MP4 VP9 playback over https with seeks landing on cue'd keyframes, and the V_VP8 unsupported-codec error.
IsDirectlyPlayable never listed .webm, so a direct WebM URL was classified as a page URL: with the resolver package installed it took a pointless yt-dlp round-trip, and without one it hard-stopped with the "looks like a page URL" error before the WebM demuxer could ever see it. The native dispatch and the resolver's extensionless googlevideo legs were unaffected — only direct .webm links through the managed router hit this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds VP9 playback end to end, using OS decoders only (no bundled codec libraries): Media Foundation
VP90on Windows (Store "VP9 Video Extensions" or a vendor MFT) and MediaCodec on Android, where every Quest hardware-decodes VP9. Part of #955.The headline user-facing win is YouTube above 1080p: the >1080p ladder is VP9-only, carried in video-only WebM, so today format selection tops out at 1080p
avc1. This PR adds:protocol/basis_webm.c) — narrow by design (first supported video track, V_VP9 only; audio stays on the existing m4a split leg). Streams clusters child-by-child rather than buffering them, tolerates any Segment-child order and unknown-size Segment/Cluster, parses all three lacing schemes defensively, and validates the EBML header limits. The CodecID switch is written so AV1 and Opus drop in as branches later.vp09-in-MP4 through both MP4 sample walks (raw samples, no Annex-B rewrite; superframes fed whole), plus MP4 track selection by role (first supported video + first supported audio, wherever they sit in the moov).basis_media_probe_video_codec()(new export, mirrored to C# asBasisNativeVideoSource.IsVideoCodecSupported). On Windows both legs must pass: the decoder MFT enumerates with exactly the flags the decode path uses, and the GPU's D3D11 video device exposes the codec's decode profile with NV12 support and a decoder configuration at the codec's offer ceiling — because the Store VP9 extension silently falls back to CPU decode on GPUs without hardware VP9, and those frames can't reach the present path. The video drain also errors loudly on the first software-decoded frame rather than showing black.avc1overvp9at equal height, so ≤1080p selection on YouTube's usual complete avc1 ladders is unchanged; machines without VP9 decode keep exactly today's behaviour.Caption scanning is gated to H.26x formats (raw VP9 bytes can contain
00 00 01runs the Annex-B walker would misparse), and stale binaries degrade cleanly (probe export missing → H.264/H.265 yes, VP9/AV1 no; missing plugin → probe false,Open()keeps its build-guidance error).Verification so far
Native-level coverage ran against dedicated fixtures on
mr.town/vod/(encoded from the CC-BY Blender films; superframe presence, Cues placements, and negative cases are structure-verified at build):vp09announce/dimensions/duration and full AU count;mp4vnegative errors correctly.Still to run before this leaves draft: the editor matrix + Windows regression rows and the WebM seek checklist from
TESTING.md, a Quest device pass, YouTube 4K end to end through the resolver, and the Store-extension-absent probe leg. Both committed binaries (win-x64, android-arm64) are RIST-enabled and were rebuilt from this tree.Required checks
All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.
TransformAccessArrayor are otherwise batched. I have not added per-frametransform.position/transform.rotation/transform.localPositioncalls inside loops. Whenever I need both position and rotation, I use the combined APIs —SetPositionAndRotation/SetLocalPositionAndRotationfor writes,GetPositionAndRotation/GetLocalPositionAndRotationfor reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.Resources.Load, no direct asset references that pull large content into memory on scene load.GetComponent/AddComponentwhere avoidable — Where unavoidable, the result is cached on a field, and anyGetComponent<T>is replaced withTryGetComponent<T>(out var x)— bareGetComponentwill be denied.TryGetComponentis the modern API (Unity 2019.2+) and skips the Editor-only GC allocationGetComponentcauses when a component is missing: Unity wraps thenullreturn in a managed "fake null" object so its overloaded==operator can still detect destroyed C++ objects, and constructing that wrapper allocates;TryGetComponentreturns aboolplusoutparameter and never builds the wrapper. None of these calls run insideUpdate,LateUpdate,FixedUpdate, jobs, or other per-frame code paths.BasisEventDriver— Any new per-frame work hooks intoBasisEventDriverrather than adding standaloneUpdate/LateUpdate/FixedUpdatecallbacks on a MonoBehaviour.BasisEventDriveris bulletproof, or guarded bytry/catch—BasisEventDriverruns the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in atry/catchthat contains the failure and surfaces it throughBasisDebug— logged once / rate-limited, never every frame (see the existingHVRBasisBuiltInAddresses.Simulate()guard for the pattern). Expect this to be scrutinized closely in review.{ get; set; }properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things offprivate/internalwithout a real reason. Don't wrap a field in{ get; set; }when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For.Instancesingletons, callers reassigningType.Instanceis allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.BasisLocalCameraDriver— Code that needs the local camera (transform, projection, rig data, etc.) pulls it fromBasisLocalCameraDriverrather than looking one up itself. Don't roll a separate camera discovery path.BasisDebug— All new logging calls go throughBasisDebug.Log/BasisDebug.LogWarning/BasisDebug.LogError(with an appropriateLogTag) instead ofUnityEngine.Debug.Log/Debug.LogWarning/Debug.LogError.BasisDebugroutes through Basis's tagged, color-coded logger and respects the project-wideLoggingDisabledtoggle so logging can be killed at runtime; bareDebug.Logcalls bypass that and will be denied.FindObjectOfType/FindObjectsOfType/GameObject.Find/FindGameObjectsWithTagto locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.newon reference types, no LINQ, nostringconcatenation/interpolation, no boxing, noforeachover interface-typed collections. Allocate once at init and reuse the buffer.BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind#if UNITY_EDITORand remove (or leave gated) before merge..Count(lists) /.Length(arrays) into a localintbefore the loop instead of re-reading the property each iteration. PreferT[](with a separate length int when the array is over-sized) overList<T>where the data is hot — Unity's mono BCL doesn't exposeCollectionsMarshal.AsSpan(List<T>), so a list can't be fed intoSpan<T>/ unsafe paths cleanly. Where the perf justifies it, drop intoSpan<T>/reflocals /Unsafe.As/unsafepointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.Testing details
Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.
Input / control mode coverage:
Where applicable, confirm these flows still work after your changes:
Notes
Editor pass, 2026-07-16 (Windows): the VP9/WebM fixture matrix from TESTING.md ran green in the editor — WebM playback with both Cues placements (inline + trailing ranged-fetch), both-direction seeks landing at/just before target, the cueless file correctly showing no seek bar, vp09-in-MP4, muxed VP9+Opus playing video with audio skipped as designed, both unsupported-codec negatives surfacing their errors, the avc1 regression control, and query-string
.webmURLs routing correctly (the direct-URL router fix in 1354f5f).Tested is unticked while this is a draft: the native layer is harness-verified as described above (that's the Windows platform tick — standalone harnesses on a Windows box, not the editor), but the editor matrix, Quest device pass, YouTube 4K end-to-end, and the Store-extension-absent probe leg haven't run yet. The box gets ticked once those pass.
Almost everything here is native C/C++ (demuxer, decode backends, probe). The C# surface is a P/Invoke, an enum, and a resolve-time format-selection change — nothing per-frame, no components, no logging, no scene access, which is why the Unity-specific checks are N/A-ticked.
Jobification: N/A — the work is native demux/decode on existing engine threads; no new managed per-frame work exists to jobify.
The probe runs on resolver worker threads by design (engine-less export, process-lifetime cached verdict, thread-safe on both backends).
TESTING.mdin both packages gained the VP9/WebM rows this PR is expected to be verified against, including the honest-seek-bar invariant (duration > 0 must always mean seeking works) and the software-fallback guard.