Drop WC_ERR_INVALID_CHARS from Windows stdio by lhecker · Pull Request #164175 · rust-lang/rust

lhecker · 2026-10-11T19:56:59Z

This PR builds on top of #164170 and refactors the
transcoding of UTF8 output to UTF16 for use with WriteConsoleW.

Previously, the stdlib would reject any non-UTF8 strings. I argue that
this is unreasonable, as historically the Windows console is a pure
character device. It makes no difference to it whether text is in an
invalid encoding, as such text is always represented as U+FFFD.
As such, it also makes no difference if we sanitize it with U+FFFD.
This is still true today, despite Windows' more modern, true PTYs.

Assuming I'm right, this PR presents a major simplification:
Output is not double-checked for validity anymore. Instead, its fed
directly into MultiByteToWideChar. THe only thing we still need
to do is splice chunked UTF8 output together, as well as we can.

This permits us to increase the buffer size from 8Ki to 64Ki wchars,
which roughly doubles throughput over the current main branch.

To make the stdout behave identical to stdin, this PR also removes
MB_ERR_INVALID_CHARS from the WideCharToMultiByte call.

Linewise output of `RELEASES.md`:

Output	Version	lines/s	MB/s	Speedup
console	1.99.0	31526	1.79
console	nightly	29251	1.66
console	#164170	67294	3.82	2.13x
console	this	73211	4.15	2.32x
`NUL`	1.99.0	784311	44.50
`NUL`	nightly	776529	44.06
`NUL`	#164170	1141618	64.78	1.46x
`NUL`	this	1129683	64.10	1.44x

128KiB chunked passthrough of `enwik8`:

Output	Version	MB/s	Speedup
console	1.99.0	86.99
console	nightly	165.27	1.90x
console	#164170	165.61	1.90x
console	this	303.51	3.49x
`NUL`	1.99.0	1123.82
`NUL`	nightly	1137.30
`NUL`	#164170	1289.35
`NUL`	this	1294.18

r? ChrisDenton

lhecker · 2026-10-11T19:58:24Z

-// Apparently Windows doesn't handle large reads on stdin or writes to stdout/stderr well (see
-// #13304 for details).
-//
-// From MSDN (2011): "The storage for this buffer is allocated from a shared heap for the
-// process that is 64 KB in size. The maximum size of the buffer will depend on heap usage."
+// The way the console used to work up to Windows 8 is that the "LPC"s carried
+// a small scratch space for reads/writes. If the client exceeded it, the server
+// allocated a larger buffer from a fixed size section object. Its size was
+// `CONSOLE_PORT_MEMORY_SIZE`, aka 64KiB. This meant any IO larger than that failed.
+// If the Windows 7 source code ever leaks: The relevant function is `ConsoleAllocateCaptureBuffer`.
 //
-// We choose the cap at 8 KiB because libuv does the same, and it seems to be acceptable so far.
+// It remains a good size for a stack allocated buffer, however.


@ChrisDenton This PR is my 2nd proposal after #164170, as such only 83ad357 is relevant here.

My proposal aside, I thought you may find this updated comment interesting. :)

View changes since the review

Indeed it is, thanks!

ChrisDenton · 2026-10-12T13:23:19Z

I think this makes a lot of sense because, aside from simplifying the code, it's much closer to our stdio behaviour on other platforms.

I'm going to nominate this for libs discussion since it would partially overturn an RFC (see https://rust-lang.github.io/rfcs/0517-io-os-reform.html#windows-and-stdio). That was an RFC from before Rust 1.0 but the given rationale for erroring does not seem convincing to me:

@rustbot label +I-libs-nominated

ChrisDenton · 2026-10-13T16:38:32Z

This was discussed in today's libs meeting. The libs team are happy for this to go ahead.

ChrisDenton

lgtm, please squish the commits then this is good to merge.

View changes since this review

lhecker · 2026-10-15T22:05:44Z

@ChrisDenton I've squashed the commits. However, I pushed a separate commit that fixes a regression in #164170, where I accidentally left a cfg() flag in. I can squash that one too, if you think that's better.

ChrisDenton · 2026-10-15T22:09:23Z

That's already being fixed by 02b3e2c which is already in the queue alongside some other win7 fixes.

lhecker · 2026-10-15T22:12:14Z

Oh, btw, I have successfully tested an experimental version of condrv.sys with buffering. Even with the lowest effort change, and no fast-pass, it results in a ~3x perf uplift when running e.g. git show/diff. In synthetic tests it ends up being up to 10x faster.
I don't know yet when I'll have time to finish it, and whether we'll get permissions to backport it.

lhecker · 2026-10-15T22:13:15Z

Ah it's great someone fixed my blunder already! I've un-pushed it.

ChrisDenton · 2026-10-15T22:28:21Z

@bors r+

Oh, btw, I have successfully tested an experimental version of condrv.sys with buffering. Even with the lowest effort change, and no fast-pass, it results in a ~3x perf uplift when running e.g. git show/diff. In synthetic tests it ends up being up to 10x faster. I don't know yet when I'll have time to finish it, and whether we'll get permissions to backport it.

That's exciting! I do hear a fair few complaints about Windows console perf so even just addressing the low hanging fruit would I think be a big win.

rust-bors · 2026-10-15T22:28:24Z

📌 Commit c5a2e72 has been tentatively approved by ChrisDenton

It will be put into the queue for this repository once PR CI succeeds.

…uwer Rollup of 7 pull requests Successful merges: - #164175 (Drop WC_ERR_INVALID_CHARS from Windows stdio) - #164361 (delegation: map return even if there is no target expression) - #162778 (rustdoc: add disambiguators for identically named method in sidebar) - #163224 ([rustdoc] Add tabs to settings popover) - #164220 (Test the interaction between `--remap-path-prefix` and doctests) - #164318 (Fix check with `is_impl_trait_in_trait()` before expecting local id) - #164360 (Fix(lib/sys): Restore Win7 compilation)

Rollup merge of #164175 - lhecker:windows-stdout-perf-chunking, r=ChrisDenton Drop WC_ERR_INVALID_CHARS from Windows stdio This PR builds on top of #164170 and refactors the transcoding of UTF8 output to UTF16 for use with `WriteConsoleW`. Previously, the stdlib would reject any non-UTF8 strings. I argue that this is unreasonable, as historically the Windows console is a pure character device. It makes no difference to it whether text is in an invalid encoding, as such text is always represented as U+FFFD. As such, it also makes no difference if we sanitize it with U+FFFD. This is still true today, despite Windows' more modern, true PTYs. Assuming I'm right, this PR presents a major simplification: Output is not double-checked for validity anymore. Instead, its fed directly into `MultiByteToWideChar`. THe only thing we still need to do is splice chunked UTF8 output together, as well as we can. This permits us to increase the buffer size from 8Ki to 64Ki wchars, which roughly doubles throughput over the current main branch. To make the stdout behave identical to stdin, this PR also removes `MB_ERR_INVALID_CHARS` from the `WideCharToMultiByte` call. ### Linewise output of `RELEASES.md`: Output | Version | lines/s | MB/s | Speedup --------|----------|--------:|------:|-------- console | 1.99.0 | 31526 | 1.79 | console | nightly | 29251 | 1.66 | console | #164170 | 67294 | 3.82 | 2.13x console | **this** | 73211 | 4.15 | 2.32x `NUL` | 1.99.0 | 784311 | 44.50 | `NUL` | nightly | 776529 | 44.06 | `NUL` | #164170 | 1141618 | 64.78 | 1.46x `NUL` | **this** | 1129683 | 64.10 | 1.44x ### 128KiB chunked passthrough of `enwik8`: Output | Version | MB/s | Speedup --------|----------|--------:|-------- console | 1.99.0 | 86.99 | console | nightly | 165.27 | 1.90x console | #164170 | 165.61 | 1.90x console | **this** | 303.51 | 3.49x `NUL` | 1.99.0 | 1123.82 | `NUL` | nightly | 1137.30 | `NUL` | #164170 | 1289.35 | `NUL` | **this** | 1294.18 |

rustbot assigned ChrisDenton Oct 11, 2026

rustbot added O-windows Operating system: Windows S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Oct 11, 2026

lhecker commented Oct 11, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

rustbot added the I-libs-nominated Nominated for discussion during a libs team meeting. label Oct 12, 2026

nia-e removed the I-libs-nominated Nominated for discussion during a libs team meeting. label Oct 13, 2026

This comment has been minimized.

Sign in to view

lhecker force-pushed the windows-stdout-perf-chunking branch from 83ad357 to 6d8b996 Compare October 15, 2026 00:38

lhecker marked this pull request as ready for review October 15, 2026 00:39

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 15, 2026

This comment has been minimized.

Sign in to view

ChrisDenton approved these changes Oct 15, 2026

View reviewed changes

Drop WC_ERR_INVALID_CHARS from Windows stdio

c5a2e72

lhecker force-pushed the windows-stdout-perf-chunking branch from 66ceffe to c5a2e72 Compare October 15, 2026 21:58

lhecker force-pushed the windows-stdout-perf-chunking branch from d9ffc65 to c5a2e72 Compare October 15, 2026 22:12

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 15, 2026

JonathanBrouwer mentioned this pull request Oct 16, 2026

Rollup of 7 pull requests #164418

Merged

rust-bors Bot merged commit d91d273 into rust-lang:main Oct 16, 2026
16 of 28 checks passed

rustbot added this to the 1.101.0 milestone Oct 16, 2026

lhecker deleted the windows-stdout-perf-chunking branch October 16, 2026 10:43

Conversation

lhecker commented Oct 11, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linewise output of RELEASES.md:

128KiB chunked passthrough of enwik8:

Uh oh!

lhecker Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChrisDenton Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ChrisDenton commented Oct 12, 2026

Uh oh!

ChrisDenton commented Oct 13, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ChrisDenton left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhecker commented Oct 15, 2026

Uh oh!

ChrisDenton commented Oct 15, 2026

Uh oh!

lhecker commented Oct 15, 2026

Uh oh!

lhecker commented Oct 15, 2026

Uh oh!

ChrisDenton commented Oct 15, 2026

Uh oh!

rust-bors Bot commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lhecker commented Oct 11, 2026 •

edited by rustbot

Loading

Linewise output of `RELEASES.md`:

128KiB chunked passthrough of `enwik8`:

lhecker Oct 11, 2026 •

edited

Loading

ChrisDenton left a comment •

edited by rustbot

Loading