panic=abort testing / subprocess testing

[Mark-Simulacrum]

This is the tracking issue for -Zpanic-abort-tests and related functionality.

Specifically, this flag switches the default test strategy of (loosely) a thread per test to a process per test, and implements the associated functionality of collecting process failures and treating that as passing/failing the test. It also includes a currently unstable option in test binaries, --force-run-in-process.

This was originally added in #64158 and Cargo support landed in rust-lang/cargo#7460.

[Mark-Simulacrum]

cc @tmandry

I would like to suggest that we change the flag to something like -Zprocess-tests or something along those lines, since AFAICT this functionality should be much more general (e.g., supporting #32512, #47506).

It's also currently a CLI flag but I would hope that we could also make this a per-test decision in panic=unwind binaries (with an attribute on the tests); panic=abort binaries would unconditionally apply it to all tests I guess.

It'd also be good to write up some docs on how to use this feature.

[tmandry]

I don't have any problem with generalizing this feature. Using -Zpanic-abort-tests is still something we haven't fully turned on by default in Fuchsia, but I'm actively working on that. I think that's a good way to uncover any issues with this approach (e.g. performance).

A related (but separate) test attribute is disallowing certain tests from being run in parallel with others. This request has come up in Fuchsia before. I believe the test harness already does this with benchmark tests.

[euclio]

This would also allow better output capturing (#42474).

[tmandry]

Update: Fuchsia has been using this feature for months now. There is one outstanding issue: #68936.

We should think about stabilizing an MVP of this feature, which is that tests work automatically when -Cpanic=abort is used. @rust-lang/libs, are there any other process steps that need to be taken first?

[dtolnay]

We should think about stabilizing an MVP of this feature, which is that tests work automatically when -Cpanic=abort is used.

I would be on board with this change. I guess send a stabilization PR?

[ehuss]

cc #73509. There's an issue with -Zpanic_abort_tests if the library has any #[bench] functions. The current implementation panics here because it tries to run the benchmark out-of-process.

It looks like for normal benchmark mode (--bench), that it just runs all benchmarks in-process. Perhaps in normal test mode, it should also run "run once" #[bench] functions in-process?

[Stargateur]

I come to an interesting problem about abort too, https://stackoverflow.com/questions/66750987/how-to-handle-abort-in-test-unit.

Could we have a #[could_abort] that ask to run a test in a process ? Or is there a better way to have a test that can abort ?

[DCNick3]

nextest seems to use separate processes for each test and (usually) can be used as a drop-in replacement for cargo test

[djkoloski]

It looks like there are two main issues blocking stabilization for -Zpanic_abort_tests so we can run tests under panic=abort:

• Tests in panic=abort can hang if test spawns a subprocess #68936
• -Zpanic_abort_tests fails with benchmarks as test #73509

Once we solve these two outstanding issues, it seems like we'd be ready to stabilize this. Are there any other issues I'm missing?