assertion failed: trying to encode a dep node twice

[zetanumbers]

As I've suspected #149916 revealed a bug in rustc's query system on multiple threads. I was trying to build rustc itself with the nightly toolchain as stage0 after doing git checkout build fails while trying to compile rustc_lint_defs.

Error

thread 'rustc' (199561) panicked at compiler/rustc_query_system/src/dep_graph/graph.rs:1392:9:
assertion `left == right` failed: trying to encode a dep node twice
  left: 64535
 right: 4294967295

backtrace

stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed::<u32, u32>
   4: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_data_structures::vec_cache::VecCache<rustc_span::def_id::LocalDefId, rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, true>
   5: rustc_query_impl::plumbing::force_from_dep_node::<rustc_query_impl::DynamicConfig<rustc_data_structures::vec_cache::VecCache<rustc_span::def_id::LocalDefId, rustc_middle::query::erase::Erased<[u8; 8]>, rustc_query_system::dep_graph::graph::DepNodeIndex>, false, false, false>>
   6: <rustc_query_impl::plumbing::query_callback<rustc_query_impl::query_impl::mir_drops_elaborated_and_const_checked::QueryType>::{closure#0} as core::ops::function::FnOnce<(rustc_middle::ty::context::TyCtxt, rustc_query_system::dep_graph::dep_node::DepNode, rustc_query_system::dep_graph::serialized::SerializedDepNodeIndex)>>::call_once
   7: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   8: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   9: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  10: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  11: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_green::<rustc_query_impl::plumbing::QueryCtxt>
  12: rustc_query_system::query::plumbing::ensure_must_run::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefIdCache<rustc_middle::query::erase::Erased<[u8; 32]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>
  13: <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners::<rustc_hir_analysis::check_crate::{closure#2}>::{closure#0}
  14: rustc_data_structures::sync::parallel::par_slice::par_rec::<&rustc_span::def_id::LocalDefId, rustc_data_structures::sync::parallel::par_for_each_in<&rustc_span::def_id::LocalDefId, &[rustc_span::def_id::LocalDefId], <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners<rustc_hir_analysis::check_crate::{closure#2}>::{closure#0}>::{closure#0}::{closure#0}>
  15: <rustc_thread_pool::job::StackJob<rustc_thread_pool::latch::SpinLatch, rustc_thread_pool::join::join_context<rustc_thread_pool::join::join::call<(), rustc_data_structures::sync::parallel::par_slice::par_rec<&rustc_span::def_id::LocalDefId, rustc_data_structures::sync::parallel::par_for_each_in<&rustc_span::def_id::LocalDefId, &[rustc_span::def_id::LocalDefId], <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners<rustc_hir_analysis::check_crate::{closure#2}>::{closure#0}>::{closure#0}::{closure#0}>::{closure#1}>::{closure#0}, rustc_thread_pool::join::join::call<(), rustc_data_structures::sync::parallel::par_slice::par_rec<&rustc_span::def_id::LocalDefId, rustc_data_structures::sync::parallel::par_for_each_in<&rustc_span::def_id::LocalDefId, &[rustc_span::def_id::LocalDefId], <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners<rustc_hir_analysis::check_crate::{closure#2}>::{closure#0}>::{closure#0}::{closure#0}>::{closure#2}>::{closure#0}, (), ()>::{closure#0}::{closure#0}, ()> as rustc_thread_pool::job::Job>::execute
  16: <rustc_thread_pool::registry::WorkerThread>::wait_or_steal_until_cold
  17: <rustc_thread_pool::registry::ThreadBuilder>::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please make sure that you have updated to the latest nightly

note: please attach the file at `/home/huawei/dev/rust/rustc-ice-2025-12-15T10_49_29-199385.txt` to your bug report

note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C debuginfo=line-tables-only -C debug-assertions=on -C incremental=[REDACTED] -C symbol-mangling-version=v0 -Z annotate-moves -Z unstable-options -Z macro-backtrace -C split-debuginfo=off -C force-frame-pointers=true -C llvm-args=-import-instr-limit=10 -C link-args=-Wl,-z,origin -C link-args=-Wl,-rpath,$ORIGIN/../lib -Z on-broken-pipe=kill -C lto=off -Z threads=24 -Z binary-dep-depinfo -Z tls-model=initial-exec -Z force-unstable-if-unmarked

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [mir_drops_elaborated_and_const_checked] elaborating drops for `<impl at compiler/rustc_lint_defs/src/lib.rs:557:1: 557:10>::default_fields_for_macro`
#1 [analysis] running analysis passes on crate `rustc_lint_defs`
end of query stack
there was a panic while trying to force a dep node
try_mark_green dep node stack:
#0 mir_for_ctfe(rustc_lint_defs[912f]::{impl#9}::default_fields_for_macro)
#1 eval_to_allocation_raw(6a497d626948f9b4-2f772ebace64724b)
#2 eval_to_const_value_raw(6a497d626948f9b4-2f772ebace64724b)
#3 eval_static_initializer(rustc_lint_defs[912f]::builtin::UNCONDITIONAL_PANIC)
end of try_mark_green dep node stack
error: could not compile `rustc_lint_defs` (lib)

It is surprisingly persistent as and I was unable to get it go away just by retriggering the incremental compilation using touch src/lib.rs but deleting incremental folder fixes this per usual after some rebuilds goes away. We can infer from the error that compiler finds itself in a situation where it tries to mark one dep node as both green and red.

Rust version

Stage 0:

rustc 1.94.0-nightly (0208ee09b 2025-12-14)
binary: rustc
commit-hash: 0208ee09be465f69005a7a12c28d5eccac7d5f34
commit-date: 2025-12-14
host: x86_64-unknown-linux-gnu
release: 1.94.0-nightly
LLVM version: 21.1.5

Steps to reproduce

Do a clean build x build library --stage 1 at 52fda2e, then git checkout to ba45f0b and without cleaning try rebuilding it the same way again. And don't forget to set RUSTFLAGS_BOOTSTRAP to -Zthreads=n where n > 1.

Run ./x b rustc in a clean git worktree of rust-lang/rust with rust.incremental = true and rust.rustflags = "-Zthreads=8" set in bootstrap.toml. Then do a change from below diff to rustc source code:

diff --git a/compiler/rustc_data_structures/src/lib.rs b/compiler/rustc_data_structures/src/lib.rs
index 4467a281118..e79312458b3 100644
--- a/compiler/rustc_data_structures/src/lib.rs
+++ b/compiler/rustc_data_structures/src/lib.rs
@@ -162,3 +162,8 @@ fn fmt(&self, f: &mut ::std::fmt::Formatter<'_>) -> ::std::fmt::Result {
         }
     };
 }
+
+#[derive(Clone, Copy)]
+pub struct BadStruct {
+    int: u64,
+}

Finally run ./x b rustc again. It may sometimes not reproduce, in such case restore changes and ./x b rustc, then apply change and ./x b rustc until ICE is reproduced.

[zetanumbers]

If I do the these steps enough times without replacing stage0 I get #141540.

[zetanumbers]

I've also got "assertion failed: tried to mark a red node as green" while messing around with the compiler. It is the same bug but it behaves a bit differently due to different orders of atomic ops. And its apparently the source of #148295.

thread 'rustc' (433110) panicked at compiler/rustc_query_system/src/dep_graph/graph.rs:1367:17:
assertion `left != right` failed: tried to mark a red node as green
  left: 4294967294
 right: 4294967294

backtrace

stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed::<u32, u32>
   4: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   5: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   6: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   7: rustc_query_system::query::plumbing::ensure_must_run::<rustc_query_impl::DynamicConfig<rustc_data_structures::vec_cache::VecCache<rustc_span::def_id::LocalDefId, rustc_middle::query::erase::Erased<[u8; 0]>, rustc_query_system::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt>
   8: <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners::<rustc_interface::passes::run_required_analyses::{closure#1}::{closure#0}>::{closure#0}
   9: rustc_data_structures::sync::parallel::par_slice::par_rec::<&rustc_span::def_id::LocalDefId, rustc_data_structures::sync::parallel::par_for_each_in<&rustc_span::def_id::LocalDefId, &[rustc_span::def_id::LocalDefId], <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners<rustc_interface::passes::run_required_analyses::{closure#1}::{closure#0}>::{closure#0}>::{closure#0}::{closure#0}>
  10: <rustc_thread_pool::job::StackJob<rustc_thread_pool::latch::SpinLatch, rustc_thread_pool::join::join_context<rustc_thread_pool::join::join::call<(), rustc_data_structures::sync::parallel::par_slice::par_rec<&rustc_span::def_id::LocalDefId, rustc_data_structures::sync::parallel::par_for_each_in<&rustc_span::def_id::LocalDefId, &[rustc_span::def_id::LocalDefId], <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners<rustc_interface::passes::run_required_analyses::{closure#1}::{closure#0}>::{closure#0}>::{closure#0}::{closure#0}>::{closure#1}>::{closure#0}, rustc_thread_pool::join::join::call<(), rustc_data_structures::sync::parallel::par_slice::par_rec<&rustc_span::def_id::LocalDefId, rustc_data_structures::sync::parallel::par_for_each_in<&rustc_span::def_id::LocalDefId, &[rustc_span::def_id::LocalDefId], <rustc_middle::ty::context::TyCtxt>::par_hir_body_owners<rustc_interface::passes::run_required_analyses::{closure#1}::{closure#0}>::{closure#0}>::{closure#0}::{closure#0}>::{closure#2}>::{closure#0}, (), ()>::{closure#0}::{closure#0}, ()> as rustc_thread_pool::job::Job>::execute
  11: <rustc_thread_pool::registry::WorkerThread>::wait_or_steal_until_cold
  12: <rustc_thread_pool::registry::ThreadBuilder>::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please make sure that you have updated to the latest nightly

note: please attach the file at `/home/huawei/dev/rust/rustc-ice-2025-12-15T12_09_14-432905.txt` to your bug report

note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C debuginfo=line-tables-only -C debug-assertions=on -C incremental=[REDACTED] -C symbol-mangling-version=v0 -Z annotate-moves -Z unstable-options -Z macro-backtrace -C split-debuginfo=off -C force-frame-pointers=true -C llvm-args=-import-instr-limit=10 -C link-args=-Wl,-z,origin -C link-args=-Wl,-rpath,$ORIGIN/../lib -Z on-broken-pipe=kill -C lto=off -Z threads=24 -Z binary-dep-depinfo -Z tls-model=initial-exec -Z force-unstable-if-unmarked

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [analysis] running analysis passes on crate `rustc_ast_pretty`
end of query stack
error: could not compile `rustc_ast_pretty` (lib)

[petrochenkov]

Issue debugged, a solution is proposed in #150156.

[luxalpa]

Are there any workarounds other than doing cargo clean? Do we know which nightly / toolchain version/date this originally appeared in?

[TeamDman]

Are there any workarounds other than doing cargo clean?

I usually

rm -r .\target\debug\incremental\

[zetanumbers]

Are there any workarounds other than doing cargo clean?

You can kinda brute-force it by doing your cargo build command again and again. At least that works when compiling rustc.

Do we know which nightly / toolchain version/date this originally appeared in?

This exact issue appeared sometime around nightly-2025-12-13. However downgrading would only get you either #142778 or #141540. The latter issue will corrupt your incremental cache and so brute-force won't be able to help you any longer.

[mischnic]

I'm also seeing these two quite frequently (nightly 2026-01-03)

thread 'rustc' panicked at /rustc-dev/7ecabfaaf1a58172cdd71f7661cf6589ddf7928f/compiler/rustc_query_system/src/dep_graph/graph.rs:1367:17:
assertion `left != right` failed: tried to mark a red node as green
  left: 4294967294
 right: 4294967294

thread 'rustc' panicked at /rustc-dev/7ecabfaaf1a58172cdd71f7661cf6589ddf7928f/compiler/rustc_query_system/src/dep_graph/graph.rs:1392:9:
assertion `left == right` failed: trying to encode a dep node twice
  left: 180797
 right: 4294967295

a few full logs from yesterday:

Archiv.zip