Bump Rust to 1.98.1 and the CI nightly to 2026-09-02

[antiguru]

Blocked: held as a draft until Rust 1.98.1 is published. The pin in this branch still reads 1.98.0 and will be moved to 1.98.1 before this is marked ready. See the note at the bottom.

CI derives its stable toolchain from the rust-version field in the root Cargo.toml, so that field is what decides which warnings CI can see. Holding it at 1.97.1 meant the lints Rust 1.98 introduced only showed up when someone built locally, which is how the warnings fixed in #38619 went unnoticed. Raising it closes that gap. Cargo.lock needs no change, which matters because the doc test job resolves with --locked.

Rust 1.98 uses LLVM 22.1.8, matching the clang-22, lld-22, and llvm-22 packages the CI builder image already installs, so the Dockerfile needs no accompanying change. The comment on that apt stanza asks for the two to move together, and they still agree. Bumping rust-version does change the builder image tag, because the tag hashes the build arguments and RUST_VERSION is one of them. ci/mkpipeline.sh detects the missing tag and inserts bootstrap steps that build and push the stable, min, and console flavors for both architectures, so the first build on this branch will be slow but needs no manual intervention.

The nightly pin moves to 2026-09-02. The note that pinned it to 2026-08-02 pointed at rust-lang/rust#160439, a rustdoc hang that broke the Doctests job, and that issue was closed as completed on 2026-08-06. The note is removed rather than reworded, because the constraint it described no longer exists.

Advancing the nightly does make rustdoc's redundant_explicit_links lint fire, and bin/doc runs with RUSTDOCFLAGS=-D warnings, so those become errors. Eight doc comments in mz-avro and mz-pgtest spell an intra-doc link as a label plus an explicit legacy HTML path that resolves to the same destination. Dropping the explicit target is the rewrite rustdoc itself suggests, and every referenced item is in scope at the link site. The flush links in the Avro writer keep their explicit targets, because a fragment path is not redundant with its label and rustdoc does not flag them.

Why this is blocked

Rust 1.98.0 carries an open P-critical miscompilation, rust-lang/rust#161441: rustc can wrongly decide an impl's predicates are impossible when they involve associated-type projections plus an opaque type, emit a vacant vtable entry, and leave a zero in the method slot. Safe code then dispatches through a null pointer. It is a regression from 1.97.1, and it is silent at compile time, so a green CI run is not evidence that the tree is unaffected.

This is not merely a developer-experience concern here, because rust-version selects the toolchain in the stable ci-builder flavor that builds the shipped images. rust-lang/rust#158993 fixed it on master on 2026-07-15, which is after the 1.98 beta cutoff and therefore too late for 1.98.0. rust-lang/rust#161555 backported it and merged to the stable branch on 2026-09-01, milestoned 1.98.1, on top of the exact 1.98.0 commit 88d9e12ae178. As of this writing 1.98.1 is not yet on the release channel.

The nightly half of this change is not exposed, since master has carried the fix since 2026-07-15 and the new pin is 2026-09-02. It is held here anyway so the toolchain moves as one reviewable change rather than two.

Outstanding before merge

• Repin Cargo.toml and bin/lint-versions to 1.98.1 once it is published, and re-run verification against it.
• bin/lint-versions records the Rust version checked for compilation time regressions. That validation has not been performed. Team Testing should confirm the final version before this merges.

The cargo test --doc job could not be exercised locally because that machine has no protoc, so it is covered only by CI. Building the nightly builder image also runs cargo miri setup and installs cargo-fuzz, neither of which can be checked outside an image build; both fail loudly in the bootstrap step rather than silently. Finally, a toolchain bump surfaces latent problems anywhere in the tree, not only in the diff, so a failure on this branch may point at code it does not touch.

Release notes

No user-visible changes.

🤖 Posted by Claude Code

[def-]

QA LLM Review

1. HIGH -- 1.98.0 is the one stable release carrying an open, unfixed UB miscompilation

Cargo.toml:271

Rust 1.98.0 miscompiles some dyn Trait calls into a null vtable slot, and this pin puts every production binary on exactly that release. It is confirmed still unfixed in 1.98.0, it is fixed in 1.98.1 (release PR merged to the stable branch on 2026-09-01) which has not been published yet, and the failure is silent at compile time so a green CI run is not evidence that the tree is unaffected.

Details

rustc 1.98.0 can decide that an impl's predicates are impossible when they involve associated-type projections plus an opaque type, emit VtblEntry::Vacant for that impl's method, and leave a zero in the method slot of a compiler-generated vtable. Safe code then dispatches through the null pointer. Upstream: introduced by rust-lang/rust#156742, reported as rust-lang/rust#161441 (P-critical, I-miscompile, regression-from-stable-to-stable, currently the only open P-critical issue), fixed on master by rust-lang/rust#158993 (merged 2026-07-15, so 1.99 beta and every nightly this PR could pin are clean), backported in rust-lang/rust#161555 for 1.98.1.

I reproduced it on the platform we build for, x86_64-unknown-linux-gnu, edition 2024, using the minimized program from the upstream issue. It compiles with no warning or error under both toolchains:

rustc 1.97.1 -O   -> prints "done", exit 0
rustc 1.98.0 -O   -> SIGILL  (exit 132), no output
rustc 1.98.0      -> SIGSEGV (exit 139), no output

The trigger needs dyn dispatch into an impl whose where-clauses involve associated-type projections or associated-type bounds, with an async fn / impl Trait in the same instantiation. Those ingredients are all present here (associated-type bounds at src/ore/src/iter.rs:235, src/compute/src/render/join/mz_join_core.rs:70, src/compute/src/render/join/linear_join.rs:112, plus pervasive async_trait erasure behind Arc<dyn ...>), so I cannot rule the tree in or out. That is the point: because rustc emits the bad vtable silently, a green build tells us nothing, and the manifestation is not limited to a crash. With optimizations on, LLVM is entitled to optimize on the assumption that the slot cannot be null, and upstream has an example of that turning into a wrong-answer behavior change rather than a fault. [profile.release] and [profile.optimized] are the profiles we ship and run.

Suggested fix: hold rust-version at 1.97.1 until 1.98.1 lands (days away, the release PR is already merged to stable), then pin 1.98.1 in Cargo.toml and bin/lint-versions. The nightly half of this PR does not need to wait, since master has carried the fix since 2026-07-15, so if the goal is to unblock the doc/nightly jobs it can land on its own. If 1.98.0 must go in before the point release, that is a decision worth stating explicitly in the commit message rather than leaving implicit in the pin.

[antiguru]

Good catch, and it holds up on every point I could check independently:

• rustc emits a vacant vtable slot for a callable boxed async service (Rust 1.98) rust-lang/rust#161441 is open, labelled P-critical, I-miscompile, regression-from-stable-to-stable, A-dyn-trait, A-impossible-bounds. The reporter confirms 1.97.1 works, 1.98.0 crashes, nightly-2026-07-16 works again.
• [stable] 1.98.1 release rust-lang/rust#161555 merged on 2026-09-01 into the stable branch, milestoned 1.98.1, backporting rerun in original typing mode if we meet any opaques in post analysis rust-lang/rust#158993. Its base sha is 88d9e12ae178fab0fb5cc050a94da85685d449ea, which is exactly the 1.98.0 release commit.
• The backport description confirms the mechanism directly: a beta cutoff fell between Add rigid alias marker rust-lang/rust#156742 and rerun in original typing mode if we meet any opaques in post analysis rust-lang/rust#158993.
• 1.98.1 is not published yet. https://static.rust-lang.org/dist/channel-rust-1.98.1.toml returns 404, and channel-rust-stable.toml still reports 1.98.0 (88d9e12ae 2026-08-18).

The point about a green CI run proving nothing is the one that decides it. rust-version selects the toolchain for the stable ci-builder flavor, which is what builds the shipped images, so this pin would reach release artifacts and not just local developer builds. I cannot demonstrate that our tree does or does not hit the pattern, which is the argument for not finding out in production when the point release is days away.

This PR is now a draft, held until 1.98.1 lands. I will then repin both Cargo.toml and bin/lint-versions to 1.98.1 and re-run verification against it.

On splitting: the nightly half is genuinely unaffected, since master has carried the fix since 2026-07-15 and the new pin is 2026-09-02. I am keeping it here rather than landing it separately, so the toolchain moves as one reviewable change. Happy to split it out instead if you would rather unblock the doc and nightly jobs now.

Two things already surfaced by the nightly half, for the record. Advancing past 2026-08-02 required removing the stale blocker note in bin/ci-builder, since rust-lang/rust#160439 was closed as completed on 2026-08-06. It also made rustdoc's redundant_explicit_links fire, which is fatal under bin/doc's -D warnings, hence the eight doc-link fixes in mz-avro and mz-pgtest.

🤖 Posted by Claude Code