Skip to content

Failure to run PGO instrumentation on Rust 1.99.0 #163616

Closed

Description

@davidhewitt

Code

I tried this code (disclosure: this was reduced from a real-world project using an AI agent, I have verified this reproduction indeed fails 1.99.0 and works fine on 1.98.1):

// repro.rs
use std::hint::black_box;

/// Reproduction needs two enum variants - `Words` has data which
/// overlaps a nested discriminant in the other variant.
enum Value {
    #[expect(dead_code)]
    Optional(Option<u64>),
    Words([u64; 2]),
}

#[inline(never)]
fn select(value: &Value) -> u64 {
    match value {
        Value::Optional(Some(_)) => 42,
        Value::Optional(None) => 7,
        Value::Words(words) => words[0],
    }
}

/// AI claims a single call to `select` incorrectly adds the `Words` payload
/// to a PGO counter on Rust 1.99.
fn main() {
    let value = black_box(Value::Words([1 << 60, 0]));
    assert_eq!(select(&value), 1 << 60);
}

I wanted to compile it and run PGO instrumentation:

rustc +1.99 -O -Cprofile-generate repro.rs -o repro
LLVM_PROFILE_FILE=repro.profraw ./repro

"$(rustc +1.99 --print target-libdir)/../bin/llvm-profdata" \
  merge -o repro.profdata repro.profraw

I expected to see this happen: PGO instrumentation runs and merges the profile successfully

Instead, this happened:

warning: repro.profraw: excessively large counter value suggests corrupted profile data: 1152921504606846976
error: no profile can be merged

Version it worked on

It most recently worked on: Rust 1.98.1

(replace +1.99 with +1.98 above in the compilation steps and it runs fine)

Version with regression

rustc --version --verbose:

rustc 1.99.0 (b940084d7 2026-09-28)
binary: rustc
commit-hash: b940084d7eb6a299eb4bfeb8e34901bc051e7ac4
commit-date: 2026-09-28
host: x86_64-unknown-linux-gnu
release: 1.99.0
LLVM version: 23.1.1

@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged

Activity

  1. added
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.
    and removed
    regression-untriagedUntriaged performance or correctness regression.
    on Oct 1, 2026
  2. lqd commented on Oct 1, 2026

    @lqd
    Member

    Does this reproduce on nightly or beta? A bisection would be helpful (https://github.com/rust-lang/cargo-bisect-rustc).

  3. added
    E-needs-bisectionCall for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc
    on Oct 1, 2026
  4. added
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    A-PGOArea: Profile-guided optimizations (PGO)
    on Oct 1, 2026
  5. Fayti1703 commented on Oct 1, 2026

    @Fayti1703
    Contributor

    I ran a bisection on this -- looks like this broke during an LLVM update (#158734).

    Raw result:

    searched nightlies: from nightly-2026-05-22 to nightly-2026-10-01
    regressed nightly: nightly-2026-08-06
    searched commit range: 1ed2df6...7608eb7
    regressed commit: 1b29cfd

    bisected with cargo-bisect-rustc v0.6.11

    Host triple: x86_64-unknown-linux-gnu
    Reproduce with:

    cargo bisect-rustc --without-cargo -c llvm-tools --start=1.97.0 --script=./check-repro.sh 

    Where check-repro.sh is:

    #!/bin/sh
    set -e
    
    rustc -O -Cprofile-generate repro.rs -o repro
    LLVM_PROFILE_FILE=repro.profraw ./repro
    "$(rustc --print target-libdir)/../bin/llvm-profdata" merge -o repro.profdata repro.profraw

    and repro.rs is (from the issue comment):

    // repro.rs
    use std::hint::black_box;
    
    /// Reproduction needs two enum variants - `Words` has data which
    /// overlaps a nested discriminant in the other variant.
    enum Value {
        #[expect(dead_code)]
        Optional(Option<u64>),
        Words([u64; 2]),
    }
    
    #[inline(never)]
    fn select(value: &Value) -> u64 {
        match value {
            Value::Optional(Some(_)) => 42,
            Value::Optional(None) => 7,
            Value::Words(words) => words[0],
        }
    }
    
    /// AI claims a single call to `select` incorrectly adds the `Words` payload
    /// to a PGO counter on Rust 1.99.
    fn main() {
        let value = black_box(Value::Words([1 << 60, 0]));
        assert_eq!(select(&value), 1 << 60);
    }
  6. added
    A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.
    S-has-bisectionStatus: A bisection has been found for this issue
    and removed
    E-needs-bisectionCall for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc
    on Oct 1, 2026
  7. theemathas commented on Oct 1, 2026

    @theemathas
    Contributor
  8. theemathas commented on Oct 1, 2026

    @theemathas
    Contributor

    By the way, thanks OP for summarizing what I assume is verbose AI output.

  9. removed
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Oct 1, 2026
  10. davidhewitt commented on Oct 1, 2026

    @davidhewitt
    ContributorAuthor

    By the way, thanks OP for summarizing what I assume is verbose AI output.

    All good; I get too many issues and PRs to PyO3 filled with AI prose and it's always extremely annoying (and a waste of my time reading the fluff), so I have no intention to ever push noise upon others. I am also aware that LLM use is extremely contentious within rust-lang, PyO3's policy is slightly more permissive but I certainly do not wish PyO3 to become slop.

    I ran a bisection on this -- looks like this broke during an LLVM update (#158734).

    For what it's worth, the agent also found a workaround in -Cllvm-args=-pgo-instr-select=false - which I assume is disabling counters for this particular case (at loss of PGO quality?).

    -Copt-level=0 also doesn't hit the problem, so I guess it's something to do with an optimization in LLVM?

    I'll see if I can turn this into something to report against LLVM.

  11. davidhewitt commented on Oct 4, 2026

    @davidhewitt
    ContributorAuthor

    I have also reported this against LLVM as llvm/llvm-project#228895

  12. self-assigned this
    on Oct 5, 2026
  13. added
    llvm-fixed-upstreamIssue expected to be fixed by the next major LLVM upgrade, or backported fixes
    on Oct 13, 2026
  14. added 2 commits that reference this issue on Oct 14, 2026
  15. linked a pull request that will close this issue[beta] backports #164371on Oct 16, 2026
  16. removed
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.A-PGOArea: Profile-guided optimizations (PGO)C-bugCategory: This is a bug.S-has-bisectionStatus: A bisection has been found for this issueT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.llvm-fixed-upstreamIssue expected to be fixed by the next major LLVM upgrade, or backported fixesregression-from-stable-to-stablePerformance or correctness regression from one stable version to another.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions