Add new `RUSTC_BOOTSTRAP=-allow-unstable-compiler-flags` mode for testing stable compiler behavior in UI tests by Kobzol · Pull Request #163664 · rust-lang/rust

Kobzol · 2026-10-02T14:39:32Z

Right now, it is not possible to test how the compiler behaves on the stable toolchain in UI tests. Even though #132993 added a mechanism to emulate stable rustc on the nightly channel, using the RUSTC_BOOTSTRAP=-1 environment variable, this mechanism is not currently usable within compiletest and UI tests directly, because compiletest unconditionally passes several unstable -Z flags that are load-bearing for the testing infra itself to work (notably -Zui-testing and friends).

That is quite annoying, because we do in fact want to test stable behavior in a couple of tests, but today we have to write those as run-make tests (latest example: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Testing.20stable.20vs.20nightly.20behavior.20in.20UI.20tests/with/628206707), even if the tested situation is very simple. Those are harder to read, write and maintain than UI tests and they do not support blessing outputs as well as UI tests. Recently, I at least added #163444 as a way of centralizing the logic of testing a stable rustc in a run-make test, but that does not really help with the fact that we still have to use run-make tests for it, even when a normal UI test would do.

For #162848, I would also like to have a way of passing -Zthreads=1 to all run-make tests, even if they are testing stable rustc, but that's not easily possible, because those tests are using RUSTC_BOOTSTRAP=-1.

This PR proposes to add a new mode to RUSTC_BOOTSTRAP, with the value of -allow-unstable-compiler-flags=<flags>. This can be used to emulate the behavior of stable rustc, while still allowing passing specific unstable flags to the compiler.

The stability check in the compiler is thus now separated into two questions:

Are we an unstable/nightly build?
Can we accept specific unstable compiler flags?

The latter is checked only at a few places, so the necessary change to the compiler is rather small (IMO).

To demonstrate how this would be used, I also implemented a new directive into compiletest in this PR, act-as-stable. That can be used in a UI test to emulate stable rustc. Some tests before used only-stable to "achieve" this, which is very misleading, because it doesn't work at all 😆, due to compiletest unconditionally passing RUSTC_BOOTSTRAP=1 to all executed tests. act-as-stable fixes that.

The last two commits demonstrate how we could replace run-make tests that tested different output on stable/nightly with UI tests.

I suppose that this mode might be interesting for other people outside of rust-lang/rust, but if you don't want that, I can gate it behind -Zui-testing.

r? jieyouxu

rustbot · 2026-10-02T14:39:36Z

compiletest directives have been modified. Please add or update docs for the
new or modified directive in src/doc/rustc-dev-guide/.

The rustc-dev-guide subtree was changed. If your future PRs only touch the subtree, consider submitting them directly to rust-lang/rustc-dev-guide, which is where the document is primarily maintained (and has faster CI).

cc @BoxyUwU, @tshepang

Some changes occurred in src/tools/compiletest

cc @jieyouxu

bjorn3 · 2026-10-02T15:09:26Z

You can probably remove

rust/compiler/rustc_codegen_cranelift/scripts/test_rustc_tests.sh

Lines 162 to 167 in c165834

    
           rm -r tests/run-make/missing-unstable-trait-bound # This disables support for unstable features, but running cg_clif needs some unstable features 
        
           rm -r tests/run-make/const-trait-stable-toolchain # same 
        
           rm -r tests/run-make/print-request-help-stable-unstable # same 
        
           rm -r tests/run-make/issue-149402-suggest-unresolve # same 
        
           rm -r tests/run-make/const-destruct-stable-toolchain # same 
        
           rm -r tests/run-make/unstable-target-feature # same

…tests

…I test

…a UI test

rustbot · 2026-10-12T06:54:45Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

jieyouxu

General idea makes sense to me. Some discussions

View changes since this review

jieyouxu · 2026-10-06T12:46:06Z

+    /// Specifically, we need it when we want to test how rustc behaves on the stable channel,
+    /// but at the same time we need to be able to pass it some unstable `-Z` compiler flags
+    /// required for the testing infrastructure itself to work.
+    ActAsStableForTesting,


I feel like this name isn't very intuitive -- if we can use unstable compiler flags then we are not acting as stable. I would directly call this AllowUnstableCompilerFlagsOnly as opposed to ActAsStableForTesting. WDYT?

In the sense that, we actually have three aspects:

Release channel: stable/beta vs nightly

Can unstable language features be used

Can unstable compiler flags be used

Ditto on the comment below

Makes sense, I'll rename it.

jieyouxu · 2026-10-12T13:09:40Z

+    // `RUSTC_BOOTSTRAP=-2` simulates the stable toolchain, while allowing unstable compiler flags
+    // to be passed.
+    std::assert_matches!(
+        UnstableFeatures::from_environment_value(None, Ok("-2".to_string())),
+        UnstableFeatures::ActAsStableForTesting
+    );


I wonder, technically I think we can call this -allow-unstable-compiler-flags right, doesn't have to be 2? Because sure --crate-name accepts a wide range of identifiers but -<whatever> is not a valid crate name.

jieyouxu · 2026-10-12T13:14:40Z

+To test stable-only behavior, use the `act-as-stable` directive, which will set `RUSTC_BOOTSTRAP=-2`
+for you.


This should also mention allow using unstable compiler flags but not unstable features

jieyouxu · 2026-10-12T13:19:31Z

 /// a best-effort approximation for diagnostics. Add new directives to this list when needed.
 pub(crate) const KNOWN_DIRECTIVE_NAMES: &[&str] = &[
    // tidy-alphabetical-start
+    "act-as-stable",


So I don't love the name, because it's actually more like //@ well-actually-we-allow-unstable-compiler-flags-but-not-language-features. But obviously that's a despairingly long name...

There's candidates like

//@ pretend-stable (which is really the same meaning as act-as-stable)

//@ stable-features-only, but that loses the "pretend" intent

Maybe //@ act-as-stable is fine.

jieyouxu · 2026-10-12T13:29:30Z

    };

-    if !edition.is_stable() && !nightly_options::is_unstable_enabled(matches) {
+    if !edition.is_stable() && !nightly_options::unstable_options_enabled(matches) {


Nit: grepping shows that there's still a reference to unstable_options_enabled around 516

View changes since the review

jieyouxu · 2026-10-12T13:34:44Z

+        handler(ACT_AS_STABLE, |_config, ln, props| {
+            if ln.name == ACT_AS_STABLE {
+                // Simulate the stable toolchain, but allow unstable -Z flags passed by compiletest
+                props.rustc_env.push(("RUSTC_BOOTSTRAP".to_string(), "-2".to_string()));
+            }
+        }),


Can we reject using this outside of ui mode as well?

(NB: I know we need to improve how we handle which directives are rejected/accepted under which conditions, but not in this PR.)

View changes since the review

petrochenkov · 2026-10-13T11:49:20Z

    let print_help = || {
        // Only show unstable options in --help if we accept unstable options.
-        let unstable_enabled = nightly_options::is_unstable_enabled(&matches);
+        let unstable_enabled = nightly_options::unstable_options_enabled(&matches);


This is actually changing the help output, I wouldn't expect the "act-as-stable" mode to do that.

FWIW, I like the "less principled escape hatch" alternative better.
Accepting all unstable flags deviates too much from acting as stable, and we don't actually need all unstable flags for UI testing, only like 2-3 of them.

View changes since the review

Yeah, if we are able to limit it to only for-testing ones, that seems generally better.

I was also thinking about that. Maybe we could do RUSTC_BOOTSTRAP=-allow-unstable-compiler-flags=ui-testing,second-flag-that-compiletest-needs, etc.? One issue with that is that compiletest actually passes also -Zunstable-options, IIRC, which increases the blast radius of the "stable-like" behavior that we change in those tests.

…mpiler-flags=<flags>`

Kobzol · 2026-10-14T11:46:08Z

I reimplemented the flag as RUSTC_BOOTSTRAP=-allow-unstable-compiler-flags=<flags>. I like this much better!

rustbot assigned jieyouxu Oct 2, 2026

Kobzol mentioned this pull request Oct 2, 2026

Use 2 parallel frontend threads by default on the nightly and dev channel #162848

Open

This comment has been minimized.

Sign in to view

Kobzol force-pushed the bootstrap-but-this-time-for-real-2 branch from 80eb9e2 to 9565273 Compare October 2, 2026 14:56

Kobzol mentioned this pull request Oct 2, 2026

Add a way for the test suite to test the output of a stable rustc #123404

Closed

This comment has been minimized.

Sign in to view

Kobzol force-pushed the bootstrap-but-this-time-for-real-2 branch from 9565273 to a36b6f8 Compare October 4, 2026 11:59

This comment has been minimized.

Sign in to view

Kobzol force-pushed the bootstrap-but-this-time-for-real-2 branch from a36b6f8 to 8e072e8 Compare October 4, 2026 17:58

This comment has been minimized.

Sign in to view

Kobzol added 5 commits October 12, 2026 08:54

Add RUSTC_BOOTSTRAP=-2 as a way of simulating a stable rustc in UI …

8240628

…tests

Implement act-as-stable compiletest directive

94a6b12

Use act-as-stable in the feature-gate-autodiff-use test

522200b

Migrate const-destruct-stable-toolchain from a run-make test to a U…

aa97b1b

…I test

Migrate print-request-help-stable-unstable from a run-make test to …

0c3b3be

…a UI test

Kobzol force-pushed the bootstrap-but-this-time-for-real-2 branch from 8e072e8 to 0c3b3be Compare October 12, 2026 06:54

petrochenkov self-assigned this Oct 12, 2026

jieyouxu reviewed Oct 12, 2026

View reviewed changes

petrochenkov reviewed Oct 13, 2026

View reviewed changes

petrochenkov removed their assignment Oct 13, 2026

jieyouxu added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 14, 2026

Replace RUSTC_BOOTSTRAP=-2 with `RUSTC_BOOTSTRAP=-allow-unstable-co…

f6c3d62

…mpiler-flags=<flags>`

Kobzol changed the title ~~Add new RUSTC_BOOTSTRAP=-2 mode for testing stable compiler behavior in UI tests~~ Add new RUSTC_BOOTSTRAP=-allow-unstable-compiler-flags mode for testing stable compiler behavior in UI tests Oct 14, 2026

This comment has been minimized.

Sign in to view

Kobzol force-pushed the bootstrap-but-this-time-for-real-2 branch from 247506c to ea39d9a Compare October 14, 2026 12:46

jieyouxu added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 14, 2026

This comment has been minimized.

Sign in to view

Use RUSTC_BOOTSTRAP=-allow-unstable-compiler-flags in compiletest

adbf2a5

Kobzol force-pushed the bootstrap-but-this-time-for-real-2 branch from ea39d9a to adbf2a5 Compare October 14, 2026 13:58

		To test stable-only behavior, use the `act-as-stable` directive, which will set `RUSTC_BOOTSTRAP=-2`
		for you.

Conversation

Kobzol commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Oct 2, 2026

Uh oh!

This comment has been minimized.

bjorn3 commented Oct 2, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rustbot commented Oct 12, 2026

Uh oh!

jieyouxu left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jieyouxu Oct 12, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jieyouxu Oct 12, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov Oct 13, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kobzol commented Oct 14, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Kobzol commented Oct 2, 2026 •

edited

Loading

jieyouxu left a comment •

edited by rustbot

Loading

jieyouxu Oct 12, 2026 •

edited by rustbot

Loading

jieyouxu Oct 12, 2026 •

edited by rustbot

Loading

petrochenkov Oct 13, 2026 •

edited by rustbot

Loading