check `extern "custom"` function pointers by folkertdev · Pull Request #159780 · rust-lang/rust

folkertdev · 2026-07-23T16:25:29Z

tracking issue: #140829
related RFC: rust-lang/rfcs#3980

Best reviewed commit-by-commit.

This PR makes 3 changes

extend the ABI checks we already performed on function definitions and trait and foreign declarations to function pointer types. This touches the various interupt ABIs and extern "custom".
remove the ability for extern "custom" to return !
improve the suggestion when safe is used in a function pointer type

folkertdev · 2026-07-23T19:00:54Z

            && match &ret_ty.kind {
-                TyKind::Never => false,
+                TyKind::Never if abi != ExternAbi::Custom => false,
                TyKind::Tup(tup) if tup.is_empty() => false,
                _ => true,
            }


@WaffleLapkin As requested, ! is now no longer allowed as a return type to extern "custom". Reading this code now, I think the ability to return ! just accidentally fell out of the logic here, and it was less of a deliberate inclusion.

We can always allow it again later, if a compelling reason comes up.

View changes since the review

rustbot · 2026-07-23T20:02:51Z

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: compiler
compiler expanded to 74 candidates
Random selection from 19 candidates

WaffleLapkin

Some nits
r? me

View changes since this review

WaffleLapkin · 2026-07-24T11:59:16Z

+        &self,
+        abi: ExternAbi,
+        ctxt: FnCtxt,
+        opt_ident: Option<&Ident>,


Could you document what the ident means/is used for?

Maybe function_name: ... + "function_name is none iff this is called for a function pointer type"?

WaffleLapkin · 2026-07-24T12:00:40Z

        self.header.ext.span().unwrap_or(self.safety_span().shrink_to_hi())
    }
+
+    pub fn as_borrowed<'a>(&'a self) -> BorrowedFnSig<'a> {


nit:

Suggested change

pub fn as_borrowed<'a>(&'a self) -> BorrowedFnSig<'a> {

pub fn as_borrowed(&self) -> BorrowedFnSig<'_> {

WaffleLapkin · 2026-07-24T12:02:49Z

+
+    pub fn as_borrowed_fn_sig<'a>(&'a self) -> BorrowedFnSig<'a> {
+        BorrowedFnSig { header: self.header(), decl: &self.decl, span: self.decl_span }
+    }


nit; this (and some other unexpected changes) is in the "update tests now that we check the ABI of function pointer types" commit, seems like something got messed up during a rebase or something...

the changes were intentional, I've updated the commit name

WaffleLapkin · 2026-07-24T12:09:53Z

            && match &ret_ty.kind {
-                TyKind::Never => false,
+                TyKind::Never if abi != ExternAbi::Custom => false,
                TyKind::Tup(tup) if tup.is_empty() => false,
                _ => true,
            }


Maybe this could be a bit more readable?

Suggested change

&& match &ret_ty.kind {

TyKind::Never => false,

TyKind::Never if abi != ExternAbi::Custom => false,

TyKind::Tup(tup) if tup.is_empty() => false,

_ => true,

}

let allowed_return = |ty| match &ret_ty.kind {

TyKind::Never if abi != ExternAbi::Custom => true,

TyKind::Tup(tup) if tup.is_empty() => true,

_ => false,

};

if let FnRetTy::Ty(ref ret_ty) = sig.decl.output

&& !allowed_return(ret_ty)

WaffleLapkin · 2026-07-24T12:16:24Z

                        // An `extern "custom"` function must be unsafe.
-                        self.reject_safe_fn(abi, ctxt, sig);
+                        self.reject_safe_fn(abi, ctxt, sig, opt_ident.is_none());


I think instead of special casing function pointers inside of reject_safe_fn, just not call it for them

For extern "custom" an error message should be generated on a missing unsafe also: extern "custom" must always be unsafe.

rustbot · 2026-07-24T12:21:44Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

WaffleLapkin · 2026-07-24T12:23:03Z

extend the ABI checks we already performed on function definitions and trait and foreign declarations to function pointer types. This touches the various interupt ABIs and extern "custom".

Are any of these stable? i.e. is this technically a breaking change we should worry about?

folkertdev · 2026-07-24T18:41:23Z

Are any of these stable?

No, they are all unstable, see:

and update the tests as needed.

rustbot · 2026-07-24T19:04:06Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

folkertdev · 2026-07-24T19:04:46Z

@rustbot ready

WaffleLapkin

@bors r+

View changes since this review

rust-bors · 2026-07-24T21:08:55Z

📌 Commit d0dde60 has been approved by WaffleLapkin

It is now in the queue for this repository.

🌲 The tree is currently closed for pull requests below priority 1. This pull request will be tested once the tree is reopened.

Reason for tree closure: spurious CI failures

…ptrs, r=WaffleLapkin check `extern "custom"` function pointers tracking issue: rust-lang#140829 related RFC: rust-lang/rfcs#3980 Best reviewed commit-by-commit. This PR makes 3 changes - extend the ABI checks we already performed on function definitions and trait and foreign declarations to function pointer types. This touches the various `interupt` ABIs and `extern "custom"`. - remove the ability for `extern "custom"` to return `!` - improve the suggestion when `safe` is used in a function pointer type

Rollup of 17 pull requests Successful merges: - #158168 (Added implementation on `set_permissions_nofollow` for all primary platforms) - #138618 (Support using const pointers in asm `const` operand) - #157962 (Function item should not be used as const arg) - #158404 (trait_solver: normalize next-gen region constraints) - #158709 (rustdoc: warn on improperly interleaved HTML/MD) - #159720 (document #[global_allocator] constraints) - #159732 (optimization: don't look for diagnostic/canonical items without rustc_attrs enabled) - #159740 (reuse regular exported_non_generic_symbols logic in Miri) - #159780 (check `extern "custom"` function pointers) - #159786 (rustdoc-js: ignore editor temp files in test folder discovery) - #159819 (std::sync::poison: disable auto_cfg on PoisonError::new) - #155388 (stepping into where-clauses during normalization may be productive) - #155914 (when bailing on ambiguity, don't force other results to ambig) - #159411 ([rustdoc] Correctly handle output options with --show-coverage) - #159439 (Fix(lib/fs/win): Fall back on Win32 delete for `Dir::remove_file`) - #159809 (Avoid `#[target_features]`) - #159826 (Remove redundant `#[rustc_paren_sugar]` feature gate)

…ptrs, r=WaffleLapkin check `extern "custom"` function pointers tracking issue: rust-lang#140829 related RFC: rust-lang/rfcs#3980 Best reviewed commit-by-commit. This PR makes 3 changes - extend the ABI checks we already performed on function definitions and trait and foreign declarations to function pointer types. This touches the various `interupt` ABIs and `extern "custom"`. - remove the ability for `extern "custom"` to return `!` - improve the suggestion when `safe` is used in a function pointer type

Rollup of 16 pull requests Successful merges: - #138618 (Support using const pointers in asm `const` operand) - #157962 (Function item should not be used as const arg) - #158404 (trait_solver: normalize next-gen region constraints) - #158709 (rustdoc: warn on improperly interleaved HTML/MD) - #159720 (document #[global_allocator] constraints) - #159732 (optimization: don't look for diagnostic/canonical items without rustc_attrs enabled) - #159740 (reuse regular exported_non_generic_symbols logic in Miri) - #159780 (check `extern "custom"` function pointers) - #159786 (rustdoc-js: ignore editor temp files in test folder discovery) - #159819 (std::sync::poison: disable auto_cfg on PoisonError::new) - #155388 (stepping into where-clauses during normalization may be productive) - #155914 (when bailing on ambiguity, don't force other results to ambig) - #159411 ([rustdoc] Correctly handle output options with --show-coverage) - #159439 (Fix(lib/fs/win): Fall back on Win32 delete for `Dir::remove_file`) - #159809 (Avoid `#[target_features]`) - #159826 (Remove redundant `#[rustc_paren_sugar]` feature gate)

…ptrs, r=WaffleLapkin check `extern "custom"` function pointers tracking issue: rust-lang#140829 related RFC: rust-lang/rfcs#3980 Best reviewed commit-by-commit. This PR makes 3 changes - extend the ABI checks we already performed on function definitions and trait and foreign declarations to function pointer types. This touches the various `interupt` ABIs and `extern "custom"`. - remove the ability for `extern "custom"` to return `!` - improve the suggestion when `safe` is used in a function pointer type

folkertdev added the F-abi_custom `#![feature(abi_custom)]` label Jul 23, 2026

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 23, 2026

folkertdev force-pushed the extern-custom-check-fn-ptrs branch 5 times, most recently from b71966b to a8807aa Compare July 23, 2026 19:41

folkertdev commented Jul 23, 2026

View reviewed changes

folkertdev marked this pull request as ready for review July 23, 2026 20:02

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 23, 2026

rustbot assigned nnethercote Jul 23, 2026

folkertdev mentioned this pull request Jul 23, 2026

Add extern "custom" rust-lang/rfcs#3980

Open

WaffleLapkin requested changes Jul 24, 2026

View reviewed changes

rustbot assigned WaffleLapkin and unassigned nnethercote Jul 24, 2026

rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 24, 2026

rustbot added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jul 24, 2026

folkertdev added 7 commits July 24, 2026 20:47

add FnHeader::span

2dc6470

make ident optional in check_extern_fn_signature

b16960f

add BorrowedFnSig to prevent a clone down the road

180f48d

add FnPtrTy::header

f7a98cb

run ABI checks on function pointer types

152ac58

and update the tests as needed.

disallow ! (i.e. Never) as a extern "custom" return type

55c729f

test extern "custom" restrictions on function pointers

a2c59e9

improve suggestion when safe is used on function pointers

d0dde60

folkertdev force-pushed the extern-custom-check-fn-ptrs branch from 5d6144e to d0dde60 Compare July 24, 2026 19:04

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 24, 2026

WaffleLapkin approved these changes Jul 24, 2026

View reviewed changes

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 24, 2026

jhpratt mentioned this pull request Jul 24, 2026

Rollup of 18 pull requests #159871

Closed

jhpratt mentioned this pull request Jul 24, 2026

Rollup of 17 pull requests #159872

Closed

jhpratt mentioned this pull request Jul 25, 2026

Rollup of 16 pull requests #159876

Closed

jhpratt mentioned this pull request Jul 25, 2026

Rollup of 19 pull requests #159879

Open

	pub fn as_borrowed<'a>(&'a self) -> BorrowedFnSig<'a> {
	pub fn as_borrowed(&self) -> BorrowedFnSig<'_> {

-            && match &ret_ty.kind {
-                TyKind::Never => false,
-                TyKind::Never if abi != ExternAbi::Custom => false,
-                TyKind::Tup(tup) if tup.is_empty() => false,
-                _ => true,
-            }
+        let allowed_return = |ty| match &ret_ty.kind {
+            TyKind::Never if abi != ExternAbi::Custom => true,
+            TyKind::Tup(tup) if tup.is_empty() => true,
+            _ => false,
+        };
+        if let FnRetTy::Ty(ref ret_ty) = sig.decl.output
+            && !allowed_return(ret_ty)

Uh oh!

Uh oh!

Conversation

folkertdev commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

folkertdev Jul 23, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Jul 23, 2026

Uh oh!

WaffleLapkin left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Jul 24, 2026

Uh oh!

WaffleLapkin commented Jul 24, 2026

Uh oh!

folkertdev commented Jul 24, 2026

Uh oh!

rustbot commented Jul 24, 2026

Uh oh!

folkertdev commented Jul 24, 2026

Uh oh!

WaffleLapkin left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rust-bors Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

folkertdev commented Jul 23, 2026 •

edited

Loading

folkertdev Jul 23, 2026 •

edited by rustbot

Loading

WaffleLapkin left a comment •

edited by rustbot

Loading

WaffleLapkin left a comment •

edited by rustbot

Loading