parallel compiler: generics_of encoding not reproducible

I tried this code:

pub struct A {}

impl A {
    pub fn f1(&self, _b1: &()) -> impl Iterator<Item = ()> + '_ {
        std::iter::empty()
    }
    pub fn f2(&self, _b2: &()) -> impl Iterator<Item = ()> + '_ {
        std::iter::empty()
    }
}

rustc --edition 2024 -Zthreads=2 --crate-type=lib --emit=metadata -o a.rmeta a.rs; md5sum a.rmeta

I expected to see this happen: equal md5sums

Instead, this happened: flipflopping md5sums

Root cause

Encodable encodes HashMap and UnordMap in iteration order. This is deterministic on a single thread, but can be nondeterministic when the parallel frontend is enabled.
In this concrete case the param_def_id_to_index field of generics_of query encoded in the rmeta is an FxHashMap

#[derive(Clone, TyEncodable, TyDecodable, StableHash)]
pub struct Generics {
    pub parent: Option<DefId>,
    pub parent_count: usize,
    pub own_params: Vec<GenericParamDef>,

    /// Reverse map to the `index` field of each `GenericParamDef`.
    #[stable_hash(ignore)]
    pub param_def_id_to_index: FxHashMap<DefId, u32>,

    pub has_self: bool,
    pub has_late_bound_regions: Option<Span>,
}

@rustbot

@rustbot label: -needs-triage +A-reproducibility +A-parallel-compiler +T-compiler

@rustbot

@rustbot claim

duplicate of #162202 (comment) ?

@aerooneqq

Might be. However there is currently work being done by @aerooneqq to fix that (see #162809). I specifically made this on his branch to showcase another issue, where we have no fix in the making. I did not test whether #162202 will be fully fixed by #162809

from reading both, dont both issues go through different things? trait impl vs inherent impl? so each would need there own fix in there own areas? alittle new to internalizing the codebase just trying to understand

@zanville

This whole class of reproducibilty errors stems from LocalDefId indices not being deterministic between runs with the parallel frontend enabled. Currently, this shows up in the generated rmeta files.

The remapping/normalization of LocalDefId-s is done in #162809, @zanville you should consider starting any new work from that branch. Here we are looking for a way to make sure any HashMap encoded in the metadata will be done in deterministic order, or for the compiler to give a warning/error when somebody attempts to write code that would encode structs containing HashMap in the metadata. That would make sure there are currently no other places than generics_of inside the metadata with HashMap inside, and helps avoid this popping up later, when metadata structs are changed.

One could consider the "simple" solution of removing Encodable from HashMap and using IndexMap instead. However other encoders, like the incremental cache, do not care about reproducibility and IndexMap has performance and memory overhead (there were no measurements done, so it might worth a try to replace them in a draft PR and do a perf run)

There is also a zulip thread for this.

I've added both cases from #162202 to #162809 so it should fix that particular example (#162202 (comment)).

However other encoders, like the incremental cache, do not care about reproducibility

I am not sure about this tbh

Here we are looking for a way to make sure any HashMap encoded in the metadata will be done in deterministic order

As a fix idea we can specialize (less preferable) or override through encoders (more preferable) encoding of all HashMaps whose keys are (local) def ids. Will be interesting to see perf.

impl<'a, 'b, T: Encodable<EncodeContext<'a, 'b>>> Encodable<EncodeContext<'a, 'b>>
    for FxHashMap<DefId, T>
{
    fn encode(&self, _: &mut EncodeContext<'a, 'b>) {
        // remap sort and encode
    }
}

impl<'a, 'b, T: Encodable<EncodeContext<'a, 'b>>> Encodable<EncodeContext<'a, 'b>>
    for FxHashMap<LocalDefId, T>
{
    fn encode(&self, _: &mut EncodeContext<'a, 'b>) {
        // remap sort and encode
    }
}

@aerooneqq

LGTM @aerooneqq, i'd like to try implementing the encoder override approach on top of
#162809 and do a perf run. is that ok with you or were you planning to add it to
#162809 yourself? happy to work around whatever you have going

Solutions for those two problems do not block each other, so you can verify locally that combination of #162809 and your changes fixes this issue, then you can create a PR with only your changes and we will run perf on it to measure its impact without #162809. Then if your changes fix this issue we can merge it ahead of #162809 (it is possible that your changes alone will solve this particular issue), if not we will merge it after #162809.

UPD: see tests that I've updated in #162809 for help in verification and feedback from #163321 about avoiding using specializations.

Sounds good I will get on that

Navigation Menu

Repository navigation

parallel compiler: generics_of encoding not reproducible #163878

Meta

Root cause

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

parallel compiler: generics_of encoding not reproducible #163878

Description

Meta

Root cause

Activity

susitsm commented on Oct 6, 2026

zanville commented on Oct 6, 2026

matthiaskrgr commented on Oct 6, 2026

susitsm commented on Oct 6, 2026

zanville commented on Oct 6, 2026

susitsm commented on Oct 7, 2026

aerooneqq commented on Oct 7, 2026

susitsm commented on Oct 7, 2026

aerooneqq commented on Oct 7, 2026

zanville commented on Oct 7, 2026

aerooneqq commented on Oct 7, 2026

zanville commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions