vectorize() + #[inline(always)] is error-prone and hard to teach

[Shnatsel]

After seeing an HPC expert who literally wrote a book on HPC and SIMD struggle with it (and entirely failing to get it to work when combined with criterion benchmarks), I'm convinced that we need to provide a more robust and convenient way to handle this.

Ideally I want to have some sort of annotation that you just put on every function generic over Simd and do not think about inlining at all, ever. The compiler just does the right thing automatically.

I see multiversion's #[multiversion(targets=simd)] (PR adding it) and thermite's #[thermite::dispatch(V)] as prior art that does this well.

AFAIK (I haven't re-checked yet) multiversion has an indirect dispatch through a cached atomic pointer on every function call, but doesn't require functions to be generic over Simd. I haven't looked into how thermite works yet.

[Shnatsel]

We could add an (optional) proc macro #[simd] in a new fearless_simd_macros crate that performs the following transformation. fearless_simd_macros would not be depended on by fearless_simd and is not subject to the v1.0 stability guarantees.

Let's say we have an input like this:

fn do_stuff<S: Simd>(simd: S, ...) {
    // do stuff here
}

This would get transformed into

fn do_stuff<S: Simd>(simd: S, ...) {
    simd.vectorize_inline(|| {
        // do stuff here
    }
}

where vectorize_inline is basically the current vectorize() but with an #[inline] on the inner function:

#[inline(always)]
fn vectorize_inline<F: FnOnce() -> R, R>(self, f: F) -> R {
    #[inline]
    #[target_feature(enable = "neon")]
    fn vectorize_neon<F: FnOnce() -> R, R>(f: F) -> R {
        f()
    }
    unsafe { vectorize_neon(f) }
}

There are drawbacks:

1. Proc macro infrastructure inflates build times
   • Kind of no way around this? Declarative macros are not as nice to use. Doing something like simd!( /* do stuff */ ) will likely run into hygiene issues. We keep the manual way for those who don't want the proc macro.
2. "The variables that a closure captures by move are dropped in an unspecified order." - Rust reference
3. Closures now show up in backtraces and on the profiler, since #[track_caller] on closures is unstable

[Shnatsel]

#347 implements this proposal

once that PR is merged, we would also need to update the docs and examples to at least mention it

[DJMcNab]

Correct me if I'm wrong, but this seems like three entirely orthogonal proposals in one:

• the first being the claim that the function calling vectorize doesn't need to be inline(always). This is presumably with the theory being that LLVM can inline a function with no target feature attributes into one with more, and then can inline a function with target features which was called by that function with fewer target features. That seems plausible to me, but I think it's a new result. The biggest risk I see with this is that the failure case is more problematic, specifically for functions which want to return values in simd registers.
• The second being that the default posture should be that we leave inlining decisions to LLVM, which also seems reasonable.
• The third being that we should use a macro to encapsulate that. Taking the second proposed point as accepted, I think I agree that it's probably the right call. When we were marking everything as #[inline(always)] with no vectorise calls, then the cost of not having a macro wasn't that high.

I also want us to be clear that this doesn't solve the other major footgun, that forgetting #[simd] is still valid but will have terrible performance.
It also has the potential to make it appear that we're on firmer footing than we actually are; as far as I know, our use of #[inline(always)] isn't guaranteed by rustc, for example.

Overall, I'm loosely in favour of this proposal, but unfortunately don't think I have time to review the PR in the near term.

[Shnatsel]

I admit I am struggling to follow your first point, but this approach does work for Vello in practice: linebender/vello#1875

I also want us to be clear that this doesn't solve the other major footgun, that forgetting #[simd] is still valid but will have terrible performance.

Yes, that is true. I have an upcoming blog post that explicitly calls this out. No library currently has a solution to this, even ones that work completely differently - e.g. multiversion. I don't see a way around this other than the Struct Target Features RFC.