Neural Rendering Compiler: Generative Video Models as Programmable Renderers
NeurCo probes a frozen video model and turns what that model can reliably do into a capability manual. A request is written as a model-agnostic program of objects, constraints, events, and cameras, then compiled into text, external operators, or both. Requirements the model already handles stay as they are. The rest are rewritten or routed to another execution strategy.
Author One* · Author Two* · Author Three · Author Four · Author Five
Institution Placeholder · *Equal contribution
NeurCo treats a frozen video model as a backend whose instruction set is not known in advance. It measures that backend once, then compiles each request against the measurement.
Offline, probe tasks run on the frozen model and are scored into a capability manual: which control, at which difficulty, under which input mode. Online, a request is split into requirements. Requirements the manual marks as supported stay unchanged. Weak ones are rewritten or handed to an external operator. The execution plan is a prompt, optional visual conditions, and operations, rendered by the same frozen model.
A. Offline capability profiling. A shared vocabulary of semantic controls is probed at graded difficulty under text-to-video, image-to-video, and reference-to-video. Each trial is judged. The manual stores the success rate and a status of ok, weak, bad, or unknown when a control was not scored.
B. Capability-guided compilation. The request becomes a model-agnostic intermediate representation of objects, constraints, events, and cameras. The compiler looks up the manual for the target model. Supported requirements stay in the original prompt. Unsupported ones are adapted only when a compatible intervention exists: a text rewrite, a reference image, a first frame, a proxy video, or shot concatenation. If none applies, the requirement is kept and left unresolved.
C. Backends and execution. One path rewrites the prompt and can retain a first frame. Another supplies a motion reference when text alone cannot carry the motion, and falls back to text if that reference does not validate. The generator stays frozen.
VComBench has 380 compositional tasks across T2V, I2V, and R2V, with 1,617 scored requirements over 34 controls. The tasks are released at Fortunaa/NeurCo-VComBench. Req. is requirement satisfaction. All is joint success on every requirement of a task. A video can satisfy some requirements and still fail the task.
![]() |
![]() |
NeurCo (T) is text-only compilation. NeurCo (T+O) also uses external operators. External PE is an external prompt enhancer.
| Method | H3 Req. / All | Wan Req. / All | Seedance 2.5 Req. / All |
|---|---|---|---|
| Native | 66.5 / 28.2 | 46.8 / 8.4 | 68.5 / 26.9 |
| NeurCo (T) | 67.7 / 27.9 | 47.9 / 8.4 | 69.7 / 27.5 |
| NeurCo (T+O) | 69.3 / 25.3 | 52.3 / 11.8 | 68.2 / 25.8 |
| External PE | 72.5 / 33.2 | 50.6 / 12.7 | 71.3 / 31.1 |
| External PE + NeurCo (O) | 73.0 / 33.7 | 52.5 / 14.5 | 72.1 / 32.2 |
| External PE + NeurCo (T) | 72.4 / 32.7 | 46.3 / 9.0 | 71.7 / 32.1 |
| External PE + NeurCo (T+O) | 72.0 / 34.5 | 50.0 / 12.3 | 71.8 / 31.5 |
On native inputs, text-only compilation is strongest for Seedance 2.5 (68.5→69.7 Req.). Wan and H3 gain more from text plus operators (46.8→52.3 and 66.5→69.3 Req.).
The project page shows a small set of Native vs NeurCo clips for H3, Wan, and Seedance 2.5. Wan reference-to-video clips are labeled R2V (VACE).
@article{neurco2026,
title = {Neural Rendering Compiler: Generative Video Models as Programmable Renderers},
author = {Author One and Author Two and Author Three and Author Four and Author Five},
journal = {arXiv preprint},
year = {2026}
}


