Skip to content

fix(hdr): send active Windows display metadata for native HDR #1048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
capy-ai merged 1 commit into from
Sep 24, 2026
Merged

fix(hdr): send active Windows display metadata for native HDR #1048

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 22 additions & 11 deletions docs/core-protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -793,26 +793,37 @@ only for this validated HDR request. `requestedStreamingFeatures.trueHdr=false`
separate because TrueHDR is the server's AI SDR-to-HDR filter, not native HDR. CloudMatch
uses bit-depth/chroma enums `1/0` for 10-bit 4:2:0 and `1/1` for 10-bit 4:4:4.
For HDR, monitor `displayData` carries validated output luminance
when the current output reports it. `desiredContentMaxLuminance` and
`desiredContentMinLuminance` come from the Wayland color-management target luminance
range in cd/m²; this mirrors the official client's feature-gated mirroring of its
system display properties into the same fields. `desiredContentMaxFrameAverageLuminance`
is omitted while a validated output snapshot is in use, because the output description
exposes no comparable sustained full-frame value and no permitted capture establishes
that mapping.
when the current output reports it. `desiredContentMaxLuminance` is the peak in nits;
`desiredContentMinLuminance` is the minimum in 0.0001-nit units, rounded after
multiplication by 10000. Wayland color-management supplies only this luminance pair.
`desiredContentMaxFrameAverageLuminance` is omitted for a pair-only snapshot because
Wayland exposes no comparable sustained full-frame value. When the runtime supplies
a complete validated optional group, the core also writes its full-frame nits into
that field and writes the red, green, blue, and white chromaticities as rounded
`xy * 50000` integers in `displayPrimaryX0/Y0`, `displayPrimaryX1/Y1`,
`displayPrimaryX2/Y2`, and `displayWhitePointX/Y`, respectively. The official PC
client's Bifrost serializer confirms the field order and scales; the mapping from
the display's maximum full-frame luminance to the frame-average field is inferred
from native struct offsets, not an observed HDR session.

Without a validated output snapshot, HDR requests keep the fixed requested-content
defaults of maximum luminance 1000 nits, minimum luminance 0, and maximum frame-average
luminance 400 nits, matching the Mac native session payload. Those defaults are requested
content characteristics rather than measurements of the physical display and are not
presented as calibration. SDR sends `displayData:null`. HDR does not invent display
primaries or a white point: the Qt output snapshot provides luminance only, so those
fields are omitted until validated chromaticities cross the Qt/core boundary.
primaries or a white point; those fields require validated chromaticities from the
current runtime output.

The validated snapshot travels from Qt in `runtimeCapabilities.nativeHdrDisplay` as
`minimumNits` and `maximumNits` in cd/m², omitting either value the output does not
report. The core validates the pair again and drops it when the bounds are not finite,
negative, above 10000 cd/m², or not strictly increasing. The snapshot is transient:
report. Optionally it also carries `maximumFullFrameNits` and normalized floating-point
`redX/redY`, `greenX/greenY`, `blueX/blueY`, and `whiteX/whiteY`. The core validates
the pair again and drops it when the bounds are not finite, negative, above 10000 cd/m²,
or not strictly increasing. It accepts the optional group only when every coordinate
is finite, within [0, 1], has a valid xy sum and positive y, the primaries form a
nondegenerate triangle, and `minimumNits < maximumFullFrameNits <= maximumNits`.
Missing or invalid optional fields discard the entire group but retain a valid pair.
The snapshot is transient:
`settings.set` rejects it, the settings loader discards persisted copies, and the core
never saves runtime capability results.

Expand Down
2 changes: 1 addition & 1 deletion docs/hdr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ the requested mode. Unsupported profiles remain explicit errors rather than CPU

## Color and ownership contract

CloudMatch receives `sdrHdrMode=1` for native HDR and `trueHdr=false`. TrueHDR requests a separate server-side AI SDR-to-HDR filter, which OpenNOW does not offer. SDR requests send `displayData:null`. HDR requests include measured output luminance when a validated snapshot is available, or requested-content defaults of 1000 nits maximum, 400 nits frame average, and zero minimum. Neither source supplies display primaries or a white point, so the core does not claim those values. Complete monitor chromaticity requires a validated output-metadata contract from the Qt shell; the luminance defaults are not physical-display measurements. The Wayland color-management target luminance range is the only output metadata either contract accepts, because a PQ description cannot report a display peak in its primary color volume luminance and the description exposes no sustained full-frame value. The accepted HDR mode is carried through resume, stream preparation, and NVST's dynamic-range and bit-depth settings.
CloudMatch receives `sdrHdrMode=1` for native HDR and `trueHdr=false`. TrueHDR requests a separate server-side AI SDR-to-HDR filter, which OpenNOW does not offer. SDR requests send `displayData:null`. HDR requests include measured output luminance when a validated snapshot is available, or requested-content defaults of 1000 nits maximum, 400 nits frame average, and zero minimum. A complete, validated runtime metadata group can additionally supply maximum full-frame luminance and display primaries and white point. Missing or invalid optional metadata is omitted rather than inferred. The luminance defaults are not physical-display measurements. Wayland color-management continues to supply only its target luminance pair: its PQ description does not establish a display peak from primary color volume luminance or expose a sustained full-frame value. The accepted HDR mode is carried through resume, stream preparation, and NVST's dynamic-range and bit-depth settings.

Decoder metadata carries transfer function, primaries, matrix, range, and supported chroma location. Explicit metadata takes precedence over negotiated defaults; unspecified fields use those defaults. Unsupported combinations and precision-losing paths fail explicitly. Pixel depth alone never selects an HDR transfer function.

Expand Down
176 changes: 166 additions & 10 deletions native/opennow-core/src/cloudmatch.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1308,13 +1308,6 @@ fn build_resume_body(app_id: &str, session: &Value, settings: &Value, device_id:
"metaData":null, "adUpdates":null})
}

fn measured_display_luminance(settings: &Value, hdr: bool) -> Option<(f64, f64)> {
if !hdr {
return None;
}
crate::streamer::validated_native_hdr_display(&settings["nativeHdrDisplay"])
}

fn monitor_display_data(hdr: bool, settings: &Value) -> Value {
if !hdr {
return Value::Null;
Expand All @@ -1324,12 +1317,34 @@ fn monitor_display_data(hdr: bool, settings: &Value) -> Value {
"desiredContentMinLuminance":0,
"desiredContentMaxFrameAverageLuminance":400
});
if let Some((minimum, maximum)) = measured_display_luminance(settings, hdr) {
data["desiredContentMaxLuminance"] = json!(maximum);
data["desiredContentMinLuminance"] = json!((minimum * 10_000.0).round() as u64);
if let Some(display) =
crate::streamer::validated_native_hdr_display(&settings["nativeHdrDisplay"])
{
data["desiredContentMaxLuminance"] = json!(display.maximum_nits);
data["desiredContentMinLuminance"] =
json!((display.minimum_nits * 10_000.0).round() as u64);
if let Some(object) = data.as_object_mut() {
object.remove("desiredContentMaxFrameAverageLuminance");
}
if let Some(metadata) = display.metadata {
data["desiredContentMaxFrameAverageLuminance"] =
json!(metadata.maximum_full_frame_nits);
for (key, coordinate) in [
"displayPrimaryX0",
"displayPrimaryY0",
"displayPrimaryX1",
"displayPrimaryY1",
"displayPrimaryX2",
"displayPrimaryY2",
"displayWhitePointX",
"displayWhitePointY",
]
.iter()
.zip(metadata.coordinates)
{
data[*key] = json!((coordinate * 50_000.0).round() as u64);
}
}
}
data
}
Expand Down Expand Up @@ -4167,6 +4182,147 @@ mod tests {
assert!(sdr_data.is_null());
}

#[test]
fn measured_monitor_metadata_survives_resolution_and_uses_wire_units() {
for peak in [400, 620, 1068] {
let full_frame = if peak == 620 { peak } else { peak - 80 };
let display = json!({"minimumNits":0.005,"maximumNits":peak,
"maximumFullFrameNits":full_frame,
"redX":0.64,"redY":0.33,"greenX":0.30,"greenY":0.60,
"blueX":0.15,"blueY":0.06,"whiteX":0.3127,"whiteY":0.329});
let capabilities = json!({"protocolVersion":7,"nativeHdrSupported":true,"videoBackends":[{
"backend":"vaapi","available":true,"codecs":[
{"codec":"h265","available":true,"colorQualities":["8bit_420","10bit_420"]}
]}],"nativeHdrDisplay":display});
let resolved = crate::streamer::StreamerService::embedded_session_settings(
&json!({"codec":"h265","enableHdr":true,"nativeHdrDisplay":{
"minimumNits":0.0,"maximumNits":1000,"maximumFullFrameNits":400,
"redX":0.7}}),
&capabilities,
)
.unwrap();
for (key, value) in display.as_object().unwrap() {
assert_eq!(
resolved["nativeHdrDisplay"][key].as_f64(),
value.as_f64(),
"{key}"
);
}
let body = build_create_body("123", &json!({}), &resolved, "device");
let request = &body["sessionRequestData"];
let data = &request["clientRequestMonitorSettings"][0]["displayData"];
assert_eq!(request["requestedStreamingFeatures"]["trueHdr"], false);
assert_eq!(
data["desiredContentMaxLuminance"].as_f64(),
Some(peak as f64)
);
assert_eq!(data["desiredContentMinLuminance"], 50);
assert_eq!(
data["desiredContentMaxFrameAverageLuminance"].as_f64(),
Some(full_frame as f64)
);
for (key, expected) in [
("displayPrimaryX0", 32000),
("displayPrimaryY0", 16500),
("displayPrimaryX1", 15000),
("displayPrimaryY1", 30000),
("displayPrimaryX2", 7500),
("displayPrimaryY2", 3000),
("displayWhitePointX", 15635),
("displayWhitePointY", 16450),
] {
assert_eq!(data[key], expected, "{key} at {peak} nits");
}
let sdr = build_create_body(
"123",
&json!({}),
&json!({"codec":"h265",
"enableHdr":false,"nativeHdrSupported":true,"nativeHdrDisplay":display}),
"device",
);
assert!(
sdr["sessionRequestData"]["clientRequestMonitorSettings"][0]["displayData"]
.is_null()
);
assert_eq!(
sdr["sessionRequestData"]["requestedStreamingFeatures"]["trueHdr"],
false
);
}
}

#[test]
fn incomplete_or_invalid_monitor_metadata_keeps_only_valid_luminance() {
let valid = json!({"minimumNits":0.005,"maximumNits":620,
"maximumFullFrameNits":400,
"redX":0.64,"redY":0.33,"greenX":0.30,"greenY":0.60,
"blueX":0.15,"blueY":0.06,"whiteX":0.3127,"whiteY":0.329});
let capabilities = |display: Value| {
json!({"protocolVersion":7,"nativeHdrSupported":true,
"videoBackends":[{"backend":"vaapi","available":true,"codecs":[
{"codec":"h265","available":true,"colorQualities":["8bit_420","10bit_420"]}
]}],"nativeHdrDisplay":display})
};
for (key, value) in [
("maximumFullFrameNits", json!(0.005)),
("maximumFullFrameNits", json!(621)),
("maximumFullFrameNits", json!(null)),
("redX", json!(1.1)),
("redY", json!(-0.1)),
("greenX", json!("0.3")),
("blueY", json!(null)),
("whiteX", json!(0.9)),
("whiteY", json!(0.0)),
("greenX", json!(0.64)),
("greenX", json!(0.640001)),
] {
let mut display = valid.clone();
display[key] = value;
if key == "blueY" {
display.as_object_mut().unwrap().remove(key);
}
if key == "greenX" && display[key] == json!(0.64) {
display["greenY"] = json!(0.33);
}
if key == "greenX" && display[key] == json!(0.640001) {
display["greenY"] = json!(0.330001);
}
let resolved = crate::streamer::StreamerService::embedded_session_settings(
&json!({"codec":"h265","enableHdr":true,"nativeHdrDisplay":valid}),
&capabilities(display),
)
.unwrap();
assert_eq!(
resolved["nativeHdrDisplay"],
json!({"minimumNits":0.005,"maximumNits":620.0}),
"{key}"
);
let body = build_create_body("123", &json!({}), &resolved, "device");
let data =
&body["sessionRequestData"]["clientRequestMonitorSettings"][0]["displayData"];
assert_eq!(data["desiredContentMaxLuminance"].as_f64(), Some(620.0));
assert_eq!(data["desiredContentMinLuminance"], 50);
assert!(data.get("desiredContentMaxFrameAverageLuminance").is_none());
assert!(data.get("displayPrimaryX0").is_none());
assert!(data.get("displayWhitePointY").is_none());
}
let no_display = crate::streamer::StreamerService::embedded_session_settings(
&json!({"codec":"h265","enableHdr":true,"nativeHdrDisplay":valid}),
&json!({"protocolVersion":7,"nativeHdrSupported":true,"videoBackends":[{
"backend":"vaapi","available":true,"codecs":[
{"codec":"h265","available":true,"colorQualities":["8bit_420","10bit_420"]}
]}]}),
)
.unwrap();
assert!(no_display.get("nativeHdrDisplay").is_none());
let fallback = build_create_body("123", &json!({}), &no_display, "device");
let data =
&fallback["sessionRequestData"]["clientRequestMonitorSettings"][0]["displayData"];
assert_eq!(data["desiredContentMaxLuminance"], 1000);
assert_eq!(data["desiredContentMinLuminance"], 0);
assert_eq!(data["desiredContentMaxFrameAverageLuminance"], 400);
}

#[test]
fn malformed_display_luminance_keeps_documented_defaults() {
for display in [
Expand Down
10 changes: 8 additions & 2 deletions native/opennow-core/src/settings.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2174,7 +2174,10 @@ mod tests {
store
.set(
"nativeHdrDisplay",
json!({"minimumNits":0.005,"maximumNits":620})
json!({"minimumNits":0.005,"maximumNits":620,
"maximumFullFrameNits":400,"redX":0.64,"redY":0.33,
"greenX":0.30,"greenY":0.60,"blueX":0.15,"blueY":0.06,
"whiteX":0.3127,"whiteY":0.329})
)
.is_err()
);
Expand All @@ -2186,7 +2189,10 @@ mod tests {
let path = directory.join("settings.json");
let mut persisted: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap();
persisted["nativeHdrSupported"] = json!(true);
persisted["nativeHdrDisplay"] = json!({"minimumNits":0.005,"maximumNits":620});
persisted["nativeHdrDisplay"] = json!({"minimumNits":0.005,"maximumNits":620,
"maximumFullFrameNits":400,"redX":0.64,"redY":0.33,
"greenX":0.30,"greenY":0.60,"blueX":0.15,"blueY":0.06,
"whiteX":0.3127,"whiteY":0.329});
fs::write(&path, serde_json::to_vec(&persisted).unwrap()).unwrap();
let mut loaded = SettingsStore::load(Some(directory.clone())).unwrap();
assert!(loaded.all().get("nativeHdrSupported").is_none());
Expand Down
65 changes: 59 additions & 6 deletions native/opennow-core/src/streamer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,10 +556,16 @@ impl StreamerService {
if let Some(object) = resolved.as_object_mut() {
object.remove("nativeHdrDisplay");
}
if let Some((minimum, maximum)) =
validated_native_hdr_display(&capabilities["nativeHdrDisplay"])
{
resolved["nativeHdrDisplay"] = json!({"minimumNits":minimum, "maximumNits":maximum});
if let Some(display) = validated_native_hdr_display(&capabilities["nativeHdrDisplay"]) {
let mut snapshot = json!({"minimumNits":display.minimum_nits,
"maximumNits":display.maximum_nits});
if let Some(metadata) = display.metadata {
snapshot["maximumFullFrameNits"] = json!(metadata.maximum_full_frame_nits);
for (key, coordinate) in HDR_CHROMATICITY_KEYS.iter().zip(metadata.coordinates) {
snapshot[*key] = json!(coordinate);
}
}
resolved["nativeHdrDisplay"] = snapshot;
}
if hdr {
resolved["colorQuality"] = json!(color);
Expand Down Expand Up @@ -1641,7 +1647,22 @@ fn apply_child_telemetry(message: &Value, state: &Arc<Mutex<Snapshot>>) {
}
}

pub(crate) fn validated_native_hdr_display(display: &Value) -> Option<(f64, f64)> {
const HDR_CHROMATICITY_KEYS: [&str; 8] = [
"redX", "redY", "greenX", "greenY", "blueX", "blueY", "whiteX", "whiteY",
];

pub(crate) struct NativeHdrDisplay {
pub minimum_nits: f64,
pub maximum_nits: f64,
pub metadata: Option<NativeHdrMetadata>,
}

pub(crate) struct NativeHdrMetadata {
pub maximum_full_frame_nits: f64,
pub coordinates: [f64; 8],
}

pub(crate) fn validated_native_hdr_display(display: &Value) -> Option<NativeHdrDisplay> {
let display = display.as_object()?;
let minimum = display.get("minimumNits")?.as_f64()?;
let maximum = display.get("maximumNits")?.as_f64()?;
Expand All @@ -1653,7 +1674,39 @@ pub(crate) fn validated_native_hdr_display(display: &Value) -> Option<(f64, f64)
{
return None;
}
Some((minimum, maximum))
let metadata = (|| {
let full_frame = display.get("maximumFullFrameNits")?.as_f64()?;
if !full_frame.is_finite() || full_frame <= minimum || full_frame > maximum {
return None;
}
let mut coordinates = [0.0; 8];
for (coordinate, key) in coordinates.iter_mut().zip(HDR_CHROMATICITY_KEYS) {
*coordinate = display.get(key)?.as_f64()?;
}
if coordinates.chunks_exact(2).any(|xy| {
!xy[0].is_finite()
|| !xy[1].is_finite()
|| !(0.0..=1.0).contains(&xy[0])
|| !(0.0..=1.0).contains(&xy[1])
|| xy[0] + xy[1] > 1.0
|| xy[1] == 0.0
}) {
return None;
}
let [rx, ry, gx, gy, bx, by, ..] = coordinates;
if ((gx - rx) * (by - ry) - (gy - ry) * (bx - rx)).abs() <= 1e-6 {
return None;
}
Some(NativeHdrMetadata {
maximum_full_frame_nits: full_frame,
coordinates,
})
})();
Some(NativeHdrDisplay {
minimum_nits: minimum,
maximum_nits: maximum,
metadata,
})
}

fn streamer_context(mut session: Value, settings: &Value) -> Value {
Expand Down
Loading
Loading