Repository navigation
mac: run bloomery on Apple Silicon (native host build, CPU path, Metal backend) #593
Description
Goal
Run bloomery on an Apple Silicon Mac: first as a native build whose host tests run, then as an engine that answers from bloomery-serve, and in the end with a Metal card path.
Today the Mac is an editor only: just mac-check and just mac-lint cross-check for x86_64 Linux, and only the pure crates run native tests.
Where it stands (counted at 8d74220bb)
| Layer | State | Size |
|---|---|---|
| Card engine | crates/gpu* is cuda-oxide throughout; the only serve::Engine over a real model is GpuModel |
386 #[kernel] fns, ~700 PTX entries (tools/ref/ptx-shapes.tsv) |
| Linux-only host APIs | sched_setaffinity/cpu_set_t (threads), RUSAGE_THREAD (gguf, engram, gpu-deepseek41), O_DIRECT (gpu, engram, levers, placement), /proc reads, hugepage advice |
~30 sites, most in gpu-gates and the labs |
| CPU kernels | qdot is AVX2; off x86 it runs the scalar mirrors (DGX Spark P0/P1) |
correct, slow; no NEON |
The DGX Spark port already did part of this: aarch64 builds, qdot off x86, and placement on unified memory (PoolOver, pool_card_bytes, docs/research/tierengine.md (h)). Apple Silicon is unified memory too, so that placement work should carry over.
Phases
- M0 — native host build. The host chain builds and tests on
aarch64-apple-darwin. The Linux-only sites go behind one platform module (affinity → no-op or QoS,RUSAGE_THREAD→ machthread_info,O_DIRECT→F_NOCACHE,/proc→sysctl). On x86 Linux nothing moves:just ptx-scanequal. - M1 — CPU path. NEON (
sdot) for the two genericqdotkernels (Q8K,Q8_2X4), then the hand kernels a host placement uses; the CPU engine (crates/model) behindserve::Engine, sobloomery-serveanswers on a Mac. - M2 — Metal backend. One device seam in
crates/gpu, CUDA and Metal behind it. Start with one fixture family's decode set (Qwen3-MoE), MSL kernels. CUDA graphs map to indirect command buffers, the fault word to a buffer atomic. - M3 — Mac gate tier. The Metal path needs gates that run on a Mac;
AGENTS.md's "gates need x86_64 Linux and an NVIDIA card" changes with it.
First step
A port map in the form of docs/contrib/dgx-spark.md: every Linux-only site with path:line and its macOS form, the kernel set one Qwen3-MoE decode step reaches, and how Metal kernels get written (hand MSL against the other options).
Activity
midagedev commentedon Oct 21, 2026
Progress, 2026-10-21.
bloomery-os. On x86 nothing moved (canon-check: 990 of 990 entries identical), and the owning gates are green.docs/contrib/mac.md(in os: the host chain builds natively on macOS; docs/contrib/mac.md, the Mac port map (#593) #600).tools/metal-probe, branchmac-probe), development readings on an M1 Pro:midagedev commentedon Oct 21, 2026
M2.0 probe on an M4 Pro (20-core GPU, 48 GB, macOS 27.0.1). Development evidence: the Mac has no timing runner yet, so these are not public numbers. Thermal state nominal at start and end, AC power. Card:
docs/cards/mac-m20-dispatch.card.Outcome:
decide-below.The chain arm prices both.
docs/contrib/mac.md: the Mac port map and design (issue #593)
os: one owner for the host chain's OS calls, so it builds natively on…
docs/contrib/mac.md: the Mac port map and design (issue #593)
os: one owner for the host chain's OS calls, so it builds natively on…
tools/metal-probe: the M2.0 probe for the per-dispatch cost and the b…
docs/contrib/mac.md: the M2.0 measurements; the chain encodes plainly…
midagedev commentedon Oct 21, 2026
Decisions on
docs/contrib/mac.mdPart F, taken by the project's rules (performance first, accuracy opt-in; common code first). Each one can be reopened when new evidence comes in.ChainOpscut over the chain walk (C.1). No Metal copy of the chain.bloomery-os.objc2-metal, pinned by version and held byjust deny. No Objective-C++ shim.metal-rsis deprecated.THIRD_PARTY_NOTICES.mdcarries the MIT notice.Open for the maintainer: 5 (who owns and reaches an M5-class machine), 6 (what reading counts as a Mac timing witness), 12 (who holds the signing and notarization keys).
M2.0 found the ICB is no faster than the plain encoder, so the ICB is dropped from the plan (
mac.mdC.5). Next measurement (M2.2a, in flight):f_q, the fraction of the stream bandwidth that a Q4_K / Q6_K mat-vec reaches on Apple GPUs.os: the host chain runs on macOS — the base page, host memory, perfor…
midagedev commentedon Oct 21, 2026
M0b landed in #611 (dd55bd9). The host crates' tests pass on macOS: base page, host memory, topology, fadvise and peak RSS now go through
bloomery-os.Found on the Mac:
hc_host::tests::hc_pre_f32_digestfails itsexp_ikdigest. Apple's libmexpfgives other bits than glibc's.exp_oursmatches. This belongs to M1 (the CPU path), not to the device track.Next: M2.1c (device contract + Metal skeleton) and M2.1d (record core lift) are in review. The M2.2a kernel probe runs on the M4 Pro tonight.
1 remaining item
midagedev commentedon Oct 21, 2026
Landed overnight:
crates/device) and its Metal backend (crates/metal). The Metal gate isjust lab-metalon a Mac.gate-deviceruns the contract's tests on the box. canon-check: 1054 of 1054 entries identical.crates/record).gpu-gatesre-exports it.AGENTS.mdanddeny.tomllines of both.Running:
ChainOpsand the qwen3moe walk inbloomery-chain. It is on top of M2.1b1, the plan move, which is not pushed yet.crates/metal, checked against the fixture's tensors. The M4 Pro checks that the port keeps the probe's speed.midagedev commentedon Oct 22, 2026
Decision (2026-10-22): kernels stay in Metal Shading Language. There is no usable layer below Metal. The GPU ISA is unpublished, AIR goes through the same driver compiler, and the Neural Engine is reachable only through CoreML with fixed graphs and shared bandwidth. On the M4 Pro our Q4_K/Q6_K mat-vecs read 229–243 GB/s on large shapes, 87–93 % of the 262 GB/s measured stream. After M2.3a the next levers are inside Metal: M5 Neural Accelerator int8/fp8 lanes for quantized prefill (no engine uses them yet; needs an M5 to measure), and MoE decode fusion.
midagedev commentedon Oct 23, 2026
@avlp12 The Metal decode path is on main now (#726). You said you got an M5 Ultra 256 GB. Could you test it?
What is there:
generate_metal.bloomery-serve-metalserves it with the llama-server API. A new chat turn reuses the previous turns through checkpoints of the delta-net state. On an M4 Pro, the second turn's time to first token went from 7.0 s to 0.63 s.Build and run:
What would help most:
llama-bench -m ... -p 512 -n 128) and mlx-lm (the 4-bit MLX build) in the same session, to compare.The Neural Engine: our M4 Pro work says the decode is bound by memory bandwidth, and the ANE is reachable only through Core ML graphs. If you see a way the ANE could help on the M5 Ultra (prefill, for example), we would like to hear it. A PR is welcome too.
Thank you!
midagedev commentedon Oct 24, 2026
MLX 4-bit files now open on Metal (#757). A first comparison on an M4 Pro (48 GB, 20-core GPU). Development evidence: the Mac has no timing runner yet, so these are not public numbers.
Model: Qwen3.6-35B-A3B. The mlx-community 4-bit folder is converted once with
prepare(19.5 GB split, about 2 minutes). The first 40 greedy tokens are the same text as mlx-lm's.Four rounds, each arm its own process, arm order rotated every round, AC power, load 2.6–4.9. Decode tok/s, mean ± SD:
102.4 ± 0.65101.6 ± 0.6487.1 ± 0.6786.4 ± 0.66bloomery on the MLX file is
+10.0 %+9.2 % over mlx-lm at the same depth (95 % interval ±1.0 %). The prompt still runs one token a step, so prompt speed is far behind mlx-lm. Batched prefill is the next Metal round.Correction (2026-10-24):
decode_tok_scounted the prompt's first token over 127 decode passes, as @avlp12 found, so our rows read 128/127 high. The struck numbers are corrected in place; the fix is #768.@avlp12, the same steps on the M5 Ultra would help a lot:
The
decode_tok_s=line on stderr is our number.avlp12 commentedon Oct 24, 2026
@midagedev Thank you for the Metal path. I ran your three checks on two machines: an M3 Ultra (512 GB, macOS 26.6.2) and an M5 Ultra (256 GB, macOS 27.0). Nothing crashed, every step exited 0, and the output was readable text, not garbage.
Setup
mainat 608c090. I built it once on the M3 Ultra and copied the two binaries to the M5 Ultra. The build had no Metal Toolchain, so the kernels compiled from source at run time.lmstudio-community/Qwen3.6-35B-A3B-GGUF,Qwen3.6-35B-A3B-Q4_K_M.gguf. Rev 68a34855, sha2564ac6a06b…, which matches the Hub.llama-benchpicked 24 threads on the M3 Ultra and 12 on the M5 Ultra; the backend was MTL,BLAS.mlx-community/Qwen3.6-35B-A3B-4bit(rev 38740b84). This is a different quantization from Q4_K_M, so the mlx-lm column is a reference, not a same-weights comparison.Commands:
1. Decode tok/s
I give two context depths so that each runtime is compared at about the same depth.
generate_metal@ d512)@ d512)*Against llama.cpp on the same GGUF,
generate_metalis 1.31–1.34x faster on the M3 Ultra and 1.36–1.40x faster on the M5 Ultra. It is also 1.21–1.33x faster than mlx-lm 4-bit.Two notes on the numbers:
generate_metal,decode_tok_scounts the prompt's first token over 127 decode steps (generate.rsstarts the clock after the prompt call). I think that makes it read about 0.8 % high.The near-0 llama.cpp numbers come from different runs on the two machines. The M3 Ultra value is from the
-d 0,512run; its-p 512 -n 128run gave 112.04 ± 1.02. The M5 Ultra value is from the clean-p 512 -n 128run; its-d 0,512run gave 158.58 ± 2.73.* These two M5 Ultra runs started while a macOS background service (
mediaanalysisd) was running at high CPU (474–487 %). Device-wide GPU utilization had a median of 21 % over 9 samples. The clean runs before it gave 156.84 (llama.cpp tg128) and 166.6 (mlx-lm), so I think the effect is small.Prompt processing
generate_metal(542 tokens)One thing I noticed: each prompt token takes 14–20 % longer than a decode token. Decode takes 6.6–6.8 ms a token on the M3 Ultra and 4.6–4.7 ms on the M5 Ultra; the low end is near depth 0 and the high end is after 542 tokens. In the source, the prompt schedule steps each id with
StepMode::HostFed(crates/metal/src/arch/qwen3moe/target.rs), which waits for each token. Decode usesStepMode::Chain, which commits the next token before it waits. If that is the cause, the prompt path could get some of this time back before batched prefill lands. I have not tested this.2. Two-turn chat against
bloomery-serve-metalI ran
-c 8192with greedy sampling andmax_tokens1024, and streamed the replies.* The M5 Ultra chat run started while
mediaanalysisdwas running. The device-wide GPU utilization median was 25 %; the first attempt, which I stopped, showedmediaanalysisdat 469 % CPU. An earlier clean run withmax_tokens64 gave a turn 1 prompt_ms of 5000 and a turn 2 of 219 ms for 23 tokens.Server log for turn 1's prompt split, then turn 2. The lines are the same on both machines:
How I read it:
Turn 2's prompt time also has a fixed cost on top of the per-token time. I fitted it from two turn-2 runs (23 and 75 tokens; the two runs used different
-c, so this is rough):The server reported
cache_msof 21–32 ms for the restore, so most of that fixed cost goes somewhere else.3. Errors and output
There were no errors. Two things about the output:
max_tokens1024 while it was still reasoning (3553 characters on both machines), so turn 2 returned an empty answer. In a first try withmax_tokens64, turn 1 spent all 64 tokens on reasoning. I don't think either of these is a bloomery problem, but I mention them in case you want to compare against your own runs.midagedev commentedon Oct 24, 2026
@avlp12 Thank you for the M3 Ultra and M5 Ultra runs. They are very useful.
decode_tok_s: you are right. It counted the prompt's first token over 127 passes. metal: generate_metal's decode rate counts the loop's passes #768 fixes it, and I corrected our M4 Pro rows above.cache_ms).max_tokens1024 is the model still reasoning, not a bloomery error.avlp12 commentedon Oct 24, 2026
@midagedev Thank you for the quick fixes. I re-ran the bloomery arms on both machines at
main0c46340.What changed and what did not
generate_metalruns, not one before each run. The device-wide GPU utilization median was 0 % on both machines.-n 128(3 runs)Prompt. The prompt is now faster by 1.15 ms an id on the M3 Ultra and 0.89 ms on the M5 Ultra. That is close to the 1.3 ms you saw on the M4 Pro. A prompt id now costs about the same as a decode step: 6.66 against 6.81 ms, and 4.55 against 4.62 ms.
Decode on the M3 Ultra. Decode is 2.0–2.4 % lower than in my first run, and the two sets of three runs do not overlap. The server's decode on the M3 Ultra fell by about 3.4 % in both chat turns: from 138.7 to 134.0, and from 136.4 to 131.8. So I think the drop is real and not only noise, but I have not found the cause. On the M5 Ultra, decode is 1.5 % lower near depth 0 and unchanged after 542 tokens. I would not use the M5 Ultra server numbers here, because the earlier M5 chat run was the one with
mediaanalysisdrunning. The prompt's first token is still in the decode count (no #768), but that bias is the same in both runs, so it cancels in these deltas. If it helps, I can rungenerate_metal --step-timeson the M3 Ultra.Turn 2 fixed cost. I estimated it the same way for both runs: turn 2's prompt_ms minus 75 ids at the
generate_metalprompt rate. That gives 153 → 164 ms on the M3 Ultra and 83 → 89 ms on the M5 Ultra. The prompt rate comes from depths 0 to 542, but turn 2 runs at depths 908 to 983. So these figures are rough, by about ±20 ms, and the cost looks about unchanged.Reuse and output. The reuse log line (
keeps 908 of 911 positions: checkpoint) is the same as before on both machines. The greedy output is the same too: turn 1's answer and the reasoning lengths match.The M5 Ultra chat runs. Both had
mediaanalysisdin the background:So the M5 chat before/after is not strictly like for like.
I can run the same set again when #768 or batched prefill lands.
metal: generate_metal's decode rate counts the loop's passes
midagedev commentedon Oct 24, 2026
@avlp12 Batched prompt prefill is merged in #773 (main b135ce1). On an M4 Pro the prompt rate went from about 100 tok/s to 760–855 tok/s (873 to 7985 ids), and decode right after a long prompt is normal again.
Could you re-run the prompt and chat TTFT numbers on the M3 Ultra and M5 Ultra at current main? Prompts of about 512 and 4096 ids next to
mlx_lm.benchmark -p 512and-p 4096would help most.One change to expect: the open now wires all weights up front, so it takes about 2–3 s longer, and the first prompt no longer pays that cost.
avlp12 commentedon Oct 24, 2026
@midagedev Congratulations on #773. I re-ran the prompt and chat numbers on both machines at
main6c15ff0, which includes #773 and #768. The setup was the same as in my earlier runs:Before every arm, the device-wide GPU utilization median was 0 % on both machines. Two other conditions differ between the machines:
mediaanalysisd) used about 360 % CPU during all arms. On the M3 Ultra it used 0 %.All three runtimes ran under those conditions on each machine.
Prompt tok/s
I fed
generate_metalseeded random ids through--tokens, with-n 128and--ctx 8192. mlx-lm and llama-bench also use random tokens. Each tool measured in its own way:generate_metal: a fresh process for each run. The 512 value is the second of two runs, and the 4096 value is the mean of two.mlx_lm.benchmark: a warmup, then the mean of 3 trials.mlx-lm uses its 4-bit MLX weights, not this GGUF.
generate_metal(batch)mlx_lm.benchmark -p(4-bit MLX)llama-benchpp (same GGUF)--prefill stepson the same 512 ids,generate_metaldid 156.0 on the M3 Ultra and 233.2 on the M5 Ultra (one run each). Batch is 11.4x and 9.5x faster.--prompt) ran at about the same rate as the random ids: 2754.7 on the M3 Ultra and 3551.4 on the M5 Ultra.generate_metal'sprompt_s(512 and 4096 ids) gives a fixed part and a per-id part:open_s: 24.2 s on the M3 Ultra and 9.0 s on the M5 Ultra. Its 512-id prompt was also slower: 1061.1 and 1404.6, against 1775.7 and 2216.8 on the next run. I don't know why the prompt was slower even though the open now wires the weights. I did not purge the page cache, so I can't say whether a disk read caused the long open.open_swas 2.66 s on the M3 Ultra and 2.54 s on the M5 Ultra. In my earlier runs it was 2.49–2.55 s and 2.39–2.49 s. I saw about 0.05–0.17 s more, not 2–3 s.Decode right after a long prompt
The two random 4096-id runs hit an end-of-generation id after 2 tokens, so I don't use their decode rate. The natural-text 4096 run decoded all 128:
generate_metalafter 4096 idsllama-benchtg128@ d4096-p 4096After 512 ids, decode was 141.0 on the M3 Ultra and 211.5 on the M5 Ultra. My previous build (0c46340, before #768) gave 143.3 and 211.8 after 542 ids. If I take out the first token that #768 stopped counting (127 of 128), those become about 142.2 and 210.1. So decode looks about unchanged.
Chat TTFT (
bloomery-serve-metal -c 8192, greedy,max_tokens1024, TTFT measured on the client)target.rsright, the source treats each part the same way:keeps 3977 of 3980 positions: checkpoint. The server now reportscheckpoints every 2048(capacity 5).--prefill stepsat the same commit, so I can't say what changed it.If it helps, I can run
--pass-timeson both machines, or repeat the M5 Ultra runs oncemediaanalysisdis idle.midagedev commentedon Oct 24, 2026
@avlp12 Thank you, this is very useful.
Yes,
--pass-timeswould help a lot. At main after #777, on both machines:It times each op of one pass on its own command buffer and prints the time per class (gemm, route, experts, delta, attention). It shows which class does not scale on the M5. The output difference you saw is expected: the batched pass uses f16 operands.
midagedev commentedon Oct 24, 2026
@avlp12 Two more PRs are merged, and one needs an M5 run.
Could you run these on the M5 Ultra, at the
mac-naxbranch (#796)?The
gemm:header should saymatrix-units=true. In thegemmlines,msoff16divided bymsoftensor16is the gain (I expect 2.5–3.5, derived, not measured). The last line printsgemm_path=tensorand the pass time by class; the same line from a main build is the before.If you have time, your 512/4096-id prompt numbers at main d854633 would help too.
avlp12 commentedon Oct 25, 2026
@midagedev Here are the
--pass-timesruns. By the time I ran them,mainhad #779, #792 and #796 in it, and on the M5 Ultra it picks the tensor-op GEMMs (gemm_path=tensor). So I measured two builds:mainbd0c613: simdgroup on the M3 Ultra, tensor on the M5 Ultra.Setup:
lmstudio-community/Qwen3.6-35B-A3B-GGUF, rev 68a34855, sha2564ac6a06b….mlx-community/Qwen3.6-35B-A3B-4bit, rev 38740b84.generate_metalsha256 is940da3b7…formainand377060bd…for 1165fd9.mediaanalysisdwas at 377–390 % CPU at every preflight sample. On the M3 Ultra it was at 0 %.I have not run your
metal_gemm_timecommands from the #796 comment yet. The M5 Ultra is busy with another GPU job right now. I will post those numbers when it is free.Pass times (
--prompt "$(cat <4096-id text>)" --ctx 8192 --pass-times U; ms, median per op, one buffer at a time)main(tensor)main(tensor)The M3 Ultra columns come from
main. On the M3 Ultra, 1165fd9 gives the same times per class, within 0.2 ms (totals 187.77 and 588.39).How I read the table:
Prompt tok/s (
generate_metal --tokens <seeded random ids> -n 128 --ctx 8192)mainHow the values were taken:
mainvalues are the mean of two runs after one unscored run.mainruns.What the numbers show:
mainvs 1165fd9. It is +25 % at 512 ids and +6 % at 4096, while the pass times are the same.mainwithin ±5 % of 1165fd9 here, so that prediction failed.prompt_sfits a smaller fixed part: about 98 ms on 1165fd9 against 50 ms onmain. The per-id part stays about 0.29 ms.--pass-timestimes the GPU per op, so it cannot see that fixed part.mainvs 1165fd9. It is 1.70x at 512 ids and 1.56x at 4096. The same fit gives about 82 → 40 ms fixed and 0.24 → 0.16 ms per id.mainis 1.07x and 1.43x on the M3 Ultra (512 and 4096 ids), and 0.87x and 1.26x on the M5 Ultra.mainis 0.97x and 1.00x on the M3 Ultra, and 0.72x and 0.59x on the M5 Ultra.mainnow scales 1.65x at 512 ids and 1.81x at 4096. My last comment had 1.25–1.29x.mainit was close to the later runs: 2450.1 against 2570.4 on the M3 Ultra, and 4169.9 against 4248.9 on the M5 Ultra. On 1165fd9, the M5 Ultra's first run was 1550.6 against 2492.7.Chat on
main(bloomery-serve-metal -c 8192, greedy,max_tokens1024, TTFT measured on the client)prefill splitline for these turns. Reuse still logskeeps 908 of 911andkeeps 3977 of 3980 positions: checkpoint.Raw
--pass-timesoutput (main, 2048 rows):midagedev commentedon Oct 25, 2026
@avlp12 감사합니다!! 앞으로 써보시면서 아쉬운점 있으면 언제든지 이슈 올려주세요!!
avlp12 commentedon Oct 25, 2026
@midagedev Thank you! I will open issues as I use it. Here are the
metal_gemm_timenumbers from the M5 Ultra that I promised for #796.Setup
metal_gemm_timeatmainbd0c613.mainright before metal: tensor-op GEMMs on the M5's neural accelerators (Metal 4 matmul2d) #796, and it includes metal: prompt call fixed costs (one buffer per layer, checkpoints copied once) #779 and metal: message marks kept inside one prompt call (prefill_marked) #792.lmstudio-community, rev 68a34855).random.Random(512)andrandom.Random(4096), the same generators as in my earlier runs.mediaanalysisdwas again at 257–388 % CPU.The header reads
gemm: architecture=applegpu_g17d metal4=true matrix-units=true engine-path=tensor.Dense (
--arm f16 --arm tensor16, gain =msof f16 /msof tensor16)The gain is 2.06–2.46x. That is a little below your 2.5–3.5 estimate. tensor16 reaches 40–68 TFLOPS, where f16 reaches 18–28.
Routed (
--routed --rows 512 --rows 2048 --arm ship --arm tensor,msof ship / tensor)generate_metal --tokens <seed-0 2048 ids> -n 1 --pass-times 2048I ran one process per build. Each value is the tool's median of 3 runs per op.
gemm_pathline printed)main(gemm_path=tensor)Delta is 11 % slower on
mainthan on d854633.crates/metal/kernels/delta.metalandsrc/kernels/delta.rsare unchanged.delta_partials,delta_prep) are no longer zeroed on the host.Prompt tok/s, d854633. I fed the same seeded random ids as in my earlier runs; each value is the mean of two runs after one unscored run.
main(09:50)These two builds were not run back to back. The ratio also covers the other commits between d854633 and
main, not only #796.