diff --git a/.agents/skills/forked-dependencies/SKILL.md b/.agents/skills/forked-dependencies/SKILL.md index 761c9e7..1531bd0 100644 --- a/.agents/skills/forked-dependencies/SKILL.md +++ b/.agents/skills/forked-dependencies/SKILL.md @@ -69,8 +69,8 @@ tqdm and mio entries. | Crate | Fork / source | Ref | Purpose | Consumed where | |-------|---------------|-----|---------|----------------| -| mistral.rs (12 crates) | `setoelkahfi/mistral.rs` | branch `fix/all-platform-fixes` | All-platform fixes; **published** as `onde-mistralrs-*` | onde deps (registry) | -| candle-core, candle-nn | `setoelkahfi/candle` | branch `onde/candle-0.11.0-27f20fea` (rev `27f20fea`) | Post-0.11.0 candle APIs; **published** as `onde-candle-{core,nn}` | mistral.rs fork deps (registry) | +| mistral.rs (12 crates) | `setoelkahfi/mistral.rs` | branch `fix/all-platform-fixes` | All-platform fixes; **published** as `onde-mistralrs-*` (0.9.3 on crates.io; 0.9.5 synced with upstream, not yet published) | onde deps (git until 0.9.5 is published, then registry) | +| candle-core, candle-nn | `setoelkahfi/candle` | branch `onde/candle-0.11.0-27f20fea` (rev `27f20fea`) | Post-0.11.0 candle APIs; **published** as `onde-candle-{core,nn}`. Upstream mistral.rs is now on candle rev `66a8cf18`, but the fork still compiles against this snapshot (host, iOS); `reduce.metal` changed in between, so watch Metal runtime behavior | mistral.rs fork deps (registry) | | tqdm | *none — upstream* | crates.io `tqdm 0.8.1` | Crossterm 0.29 (→ mio 1.x) is upstream as of 0.8.1 | mistral.rs fork dep (registry) | | sysctl | `setoelkahfi/sysctl-rs` | branch `feature/watchos` | watchOS (`target_os = "watchos"`) support | `[patch.crates-io]` in onde **and** mistral.rs fork | | mio | `setoelkahfi/mio` | branch `feature/visionos` | visionOS support for the `mio 0.8.11` line | **RETIRED** — patch removed from onde once it consumed onde-mistralrs 0.9.3 (crossterm 0.29 → mio 1.x). Fork kept for history/upstream PR. | @@ -98,6 +98,13 @@ tqdm and mio entries. `-flash-attn`, `-metal-compile`, `-code-exec`, `-sandbox`, `-macros`, `-vision`, `-audio`, `-mcp`. The rename is a `package = "onde-..."` override on each workspace dep entry, so Rust source still `use`s `mistralrs_*` unchanged. +- **Paged attention:** upstream #2493 made Metal paged attention the default. + onde turns it off on iOS, tvOS, visionOS and watchOS with + `.without_paged_attn()` (GGUF and UQFF builders in `engine.rs`, and the + validation load in `hf_cache.rs`); macOS keeps the default. Android is CPU + and unchanged. +- **0.9.5 API change:** `with_device_mapping` now takes `DeviceMapMetadata`, + so the tvOS memory cap uses `.with_max_model_len(512)` instead. - **Publishing + bump flow:** see [[sdk-regressions]] (the 12-crate publish order, the candle Step 0, and the tqdm/core2 publish-verification snags). diff --git a/.agents/skills/sdk-regressions/SKILL.md b/.agents/skills/sdk-regressions/SKILL.md index 9ee49fc..b67a11a 100644 --- a/.agents/skills/sdk-regressions/SKILL.md +++ b/.agents/skills/sdk-regressions/SKILL.md @@ -422,7 +422,7 @@ target first. ### Step 0 — check if the candle snapshot needs an `onde-candle-*` publish Since 0.9.x, upstream mistral.rs pins `candle` to a git rev past the latest -released candle (e.g. rev `27f20fea`, 19 commits past `0.11.0`, adding +released candle (e.g. rev `27f20fea`; upstream has since moved to `66a8cf18`, which 0.9.5 does not need to compile, 19 commits past `0.11.0`, adding `QTensor::indexed_gemv`, `gemv_fused_shared_lhs`, `BarrierPool::execute_chunked`). `cargo publish` strips git pins, so publish verification resolves candle from crates.io and fails with E0599 "method not found" if the pinned rev has APIs the @@ -456,6 +456,18 @@ candle-metal-kernels = { version = "0.11.0" } # unchanged at rev -> official ### Step 1 — check if `onde-mistralrs` needs a new publish +> Testing before publish: keep onde on the registry form +> (`version = "0.9.5", package = "onde-mistralrs{,-core}"`) and patch by the real +> package name from the command line, e.g. `cargo check --config +> 'patch.crates-io.onde-mistralrs.git="https://github.com/setoelkahfi/mistral.rs"' +> --config 'patch.crates-io.onde-mistralrs.branch="fix/all-platform-fixes"'` +> (repeat for `onde-mistralrs-core`). A `[patch.crates-io]` rename of `mistralrs` +> is silently ignored. `cargo package -p ` in the fork verifies +> the packaged crates against an overlay of their unpublished siblings. Don't +> commit the Cargo.lock these produce. Since 0.9.5, Metal +> paged attention is upstream's default (#2493), so onde calls +> `.without_paged_attn()` on iOS/tvOS/visionOS/watchOS. + First, see how far the fork has drifted from upstream: ```bash diff --git a/CHANGELOG.md b/CHANGELOG.md index 6545fb7..09acfaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,16 @@ +## 1.3.4 + +### Inference engine + +* Moves `onde-mistralrs` and `onde-mistralrs-core` from `0.9.3` to `0.9.5` on every target. The new crates merge upstream mistral.rs `0.9.5` into the fork and add visionOS and watchOS arms to the Metal `embedded_library!` macro, so `--features metal` builds for tvOS, visionOS and watchOS again. +* Upstream made Metal paged attention the default. Its KV block pool does not fit in memory on iOS, tvOS, visionOS or watchOS, so model loading (and the validation load in `hf_cache`) now calls `without_paged_attn()` on those targets. Desktop and Android are unchanged. +* On tvOS the auto device map override is replaced by `with_max_model_len(512)`. + +### Packaging + +* No public API changes: the published `onde` crate surface is identical to `1.3.3`. +* All version sources (root, Kotlin, Dart, React Native, and the root and per-SDK lockfiles) are aligned on `1.3.4`. + ## 1.3.3 ### Cargo workspace diff --git a/Cargo.lock b/Cargo.lock index eb13a5d..a96d33d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1318,17 +1318,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "derive-new" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cdc8d50f426189eef89dac62fabfa0abb27d5cc008f25bf4156a0203325becc" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "derive_builder" version = "0.20.2" @@ -3318,6 +3307,16 @@ dependencies = [ "paste", ] +[[package]] +name = "metrics" +version = "0.24.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89550ee9f79e88fef3119de263694973a8adb26c21d75322164fb8c493039fe2" +dependencies = [ + "portable-atomic", + "rapidhash", +] + [[package]] name = "mime" version = "0.3.17" @@ -3340,6 +3339,7 @@ version = "2.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "86886cf6dbf4e614b19c9a1eec9775f021869d7eadde0fc73921a81b90c9b4c9" dependencies = [ + "indexmap 2.14.2", "memo-map", "serde", "serde_json", @@ -3729,7 +3729,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" [[package]] name = "onde" -version = "1.3.3" +version = "1.3.4" dependencies = [ "anyhow", "cc", @@ -3799,9 +3799,9 @@ dependencies = [ [[package]] name = "onde-mistralrs" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "689f038ebc5d6d11e4880a24e50d62f91bbf21358b92c9b5cf7bf5e65af232bd" +checksum = "0d9f41269feaffb3e452022f04a9cfb95859ed950889b6c1d010391531847260" dependencies = [ "anyhow", "clap 4.6.6", @@ -3827,9 +3827,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-audio" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71b9a72504697c5b91e1d0448d67883e8d3046ad060200610c0484aef5dfaee5" +checksum = "787f40176eda80bdef527227ba43a7d4dac3287a3d7e28af5b4072b92abee042" dependencies = [ "anyhow", "apodize", @@ -3839,9 +3839,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-code-exec" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4016265a19fa51ad973e99a6746904e432d3140b5c2ca14a41077edc6b8648c2" +checksum = "b3851924d7b5dfceaa8309ada9d04d983a4b6cf8761117145e1a035011e4078f" dependencies = [ "anyhow", "base64 0.22.1", @@ -3859,9 +3859,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-core" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3e9266eb62ceb5a9cf8fb5342e6a9b78e286f47b670762559ee800a5a2db5132" +checksum = "95d792d7cb5a4b78f4a448aa246bb6fbb8ea84a7958ce256173c12a83adbafb1" dependencies = [ "ahash", "akin", @@ -3878,9 +3878,9 @@ dependencies = [ "chrono", "clap 4.6.6", "csv", - "derive-new", "derive_more", "dirs", + "dispatch2", "either", "float8", "futures", @@ -3899,6 +3899,7 @@ dependencies = [ "libc", "llguidance", "lrtable", + "metrics", "mime_guess", "minijinja", "minijinja-contrib", @@ -3910,11 +3911,11 @@ dependencies = [ "onde-mistralrs-audio", "onde-mistralrs-code-exec", "onde-mistralrs-mcp", + "onde-mistralrs-metal-compile", "onde-mistralrs-paged-attn", "onde-mistralrs-quant", "onde-mistralrs-sandbox", "onde-mistralrs-vision", - "onde-tqdm", "openai-harmony", "ordered-float", "parking_lot", @@ -3938,10 +3939,12 @@ dependencies = [ "serde-saphyr", "serde_json", "serde_plain", + "sha2", "statrs", "strum 0.27.2", "symphonia", "sysinfo", + "tempfile", "thiserror 2.0.20", "tokenizers 0.21.4", "tokio", @@ -3950,6 +3953,7 @@ dependencies = [ "toktrie", "toktrie_hf_tokenizers", "toml", + "tqdm", "tracing", "tracing-subscriber", "urlencoding", @@ -3960,9 +3964,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-macros" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e624ab1f0896a07ff437f4c49050c891918c0f01a008113c788b64e5f76c1d9" +checksum = "8dd60c4cb0dc37105b335451643d075af0d293b8203df9bb038e88d83df8e475" dependencies = [ "darling 0.23.0", "proc-macro2", @@ -3972,9 +3976,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-mcp" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8bcb140ed84c4dd2683f1533fc58905f959e7394843bc02a315bfdd3ffcbc25" +checksum = "f5b5bf129c896c975a98575b9752291bb47b38a5bd3170944a8c171cc491af1b" dependencies = [ "anyhow", "async-trait", @@ -3994,9 +3998,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-metal-compile" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "44109415be2971d311b403095d3c278443210d0ccc1c383a84ad44da5f5b1abf" +checksum = "02e15b1154394bc9750f69c52b55de0295e6f7d3a3f3151f9ad8d88e066629af" dependencies = [ "candle-metal-kernels", "objc2-metal", @@ -4004,9 +4008,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-paged-attn" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f93c8c04d15ebce014e80675387d269bcead85be063bf9afce8b353d64e61b8" +checksum = "9ba6ae15cc8ce43c949fd038a8df0858e41dca38e4b4b77925aa04e6dc4aee84" dependencies = [ "anyhow", "candle-metal-kernels", @@ -4022,17 +4026,20 @@ dependencies = [ [[package]] name = "onde-mistralrs-quant" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d4ed48af14e3e5ff318a6ea6cefbc1e50ee753f143bd1ae6c0f188c33033d5a" +checksum = "a25f0259f21cf04d86f18b3b0f48b072bcc4a989cadae513ca142ab1ccb0c60a" dependencies = [ + "bytemuck", "byteorder", "candle-metal-kernels", "dispatch2", + "fancy-regex 0.14.0", "float8", "futures", "half", "hf-hub", + "indexmap 2.14.2", "indicatif 0.18.6", "lazy_static", "libc", @@ -4057,9 +4064,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-sandbox" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77845cc6e16e3b1eebed11d3ba6a3a56785f0031421bf46d1deefbe281aaa19e" +checksum = "93b8ba017e502f2b7f2da67ced422ae73a7ea2494759b8c7a8e5fa38b341b3f5" dependencies = [ "anyhow", "itoa", @@ -4075,9 +4082,9 @@ dependencies = [ [[package]] name = "onde-mistralrs-vision" -version = "0.9.3" +version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8cdc7ae244c416860838e9cb3ff5cd8ae8acf474f48f2fe15d45bc5491215760" +checksum = "f231953141403eb6a825d6c247dffd7aa2e99b01b7a70d075bb72533c8dc10e8" dependencies = [ "image", "onde-candle-core", @@ -4086,7 +4093,7 @@ dependencies = [ [[package]] name = "onde-react-native" -version = "1.3.3" +version = "1.3.4" dependencies = [ "jni 0.21.1", "log", @@ -4096,17 +4103,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "onde-tqdm" -version = "0.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d96f4fcb4af06959bf9c9b81ace0502e2be2d11d7e037c94f3f7851bde720ea" -dependencies = [ - "anyhow", - "crossterm", - "once_cell", -] - [[package]] name = "onde-uniffi-bindgen" version = "0.1.0" @@ -4710,6 +4706,15 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "rapidhash" +version = "4.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5da7e78a036ce858e8d55b7e7dc8ba3a88b78350fd2155d3591bbd966b58589e" +dependencies = [ + "rustversion", +] + [[package]] name = "rav1e" version = "0.8.1" @@ -6666,6 +6671,17 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" +[[package]] +name = "tqdm" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a882329b482cea1d2762e9b9ff1ed0f0327a1ec3a9ba7b8e8c4e3635908a4ae5" +dependencies = [ + "anyhow", + "crossterm", + "once_cell", +] + [[package]] name = "tracing" version = "0.1.44" diff --git a/Cargo.toml b/Cargo.toml index a013a83..d40db41 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "onde" -version = "1.3.3" +version = "1.3.4" edition = "2021" description = "On-device inference engine for Apple silicon." license = "MIT OR Apache-2.0" @@ -45,51 +45,51 @@ crate-type = ["lib", "cdylib", "staticlib"] # in the Android sandbox). [target.'cfg(target_os = "android")'.dependencies] hf-hub = { version = "0.4", features = ["tokio", "rustls-tls"], default-features = false } -mistralrs = { version = "0.9.3", package = "onde-mistralrs", features = [] } -mistralrs-core = { version = "0.9.3", package = "onde-mistralrs-core", features = [] } +mistralrs = { version = "0.9.5", package = "onde-mistralrs", features = [] } +mistralrs-core = { version = "0.9.5", package = "onde-mistralrs-core", features = [] } # iOS: Metal-accelerated inference; hf-hub + mistralrs-core needed for # GLOBAL_HF_CACHE sandbox workaround (~/.cache is outside the container). [target.'cfg(target_os = "ios")'.dependencies] hf-hub = { version = "0.4", features = ["tokio", "rustls-tls"], default-features = false } -mistralrs = { version = "0.9.3", package = "onde-mistralrs", features = ["metal"] } -mistralrs-core = { version = "0.9.3", package = "onde-mistralrs-core", features = ["metal"] } +mistralrs = { version = "0.9.5", package = "onde-mistralrs", features = ["metal"] } +mistralrs-core = { version = "0.9.5", package = "onde-mistralrs-core", features = ["metal"] } # Linux: CPU-only baseline (no GPU features). # For NVIDIA GPU builds, patch this to features = ["cuda"] in CI before building. [target.'cfg(target_os = "linux")'.dependencies] -mistralrs = { version = "0.9.3", package = "onde-mistralrs" } +mistralrs = { version = "0.9.5", package = "onde-mistralrs" } # macOS: Metal-accelerated inference. [target.'cfg(target_os = "macos")'.dependencies] -mistralrs = { version = "0.9.3", package = "onde-mistralrs", features = ["metal"] } +mistralrs = { version = "0.9.5", package = "onde-mistralrs", features = ["metal"] } # tvOS: Metal-accelerated inference (Apple TV has Apple GPU); hf-hub + # mistralrs-core needed for GLOBAL_HF_CACHE sandbox workaround. [target.'cfg(target_os = "tvos")'.dependencies] hf-hub = { version = "0.4", features = ["tokio", "rustls-tls"], default-features = false } -mistralrs = { version = "0.9.3", package = "onde-mistralrs", features = ["metal"] } -mistralrs-core = { version = "0.9.3", package = "onde-mistralrs-core", features = ["metal"] } +mistralrs = { version = "0.9.5", package = "onde-mistralrs", features = ["metal"] } +mistralrs-core = { version = "0.9.5", package = "onde-mistralrs-core", features = ["metal"] } # visionOS: Metal-accelerated inference (Apple Vision Pro has M2 GPU); # hf-hub + mistralrs-core needed for GLOBAL_HF_CACHE sandbox workaround. [target.'cfg(target_os = "visionos")'.dependencies] hf-hub = { version = "0.4", features = ["tokio", "rustls-tls"], default-features = false } -mistralrs = { version = "0.9.3", package = "onde-mistralrs", features = ["metal"] } -mistralrs-core = { version = "0.9.3", package = "onde-mistralrs-core", features = ["metal"] } +mistralrs = { version = "0.9.5", package = "onde-mistralrs", features = ["metal"] } +mistralrs-core = { version = "0.9.5", package = "onde-mistralrs-core", features = ["metal"] } # watchOS: Metal-accelerated inference (Apple Watch Series 4+ has Apple GPU); # hf-hub + mistralrs-core needed for GLOBAL_HF_CACHE sandbox workaround. # Memory constrained — default to 1.5B model. [target.'cfg(target_os = "watchos")'.dependencies] hf-hub = { version = "0.4", features = ["tokio", "rustls-tls"], default-features = false } -mistralrs = { version = "0.9.3", package = "onde-mistralrs", features = ["metal"] } -mistralrs-core = { version = "0.9.3", package = "onde-mistralrs-core", features = ["metal"] } +mistralrs = { version = "0.9.5", package = "onde-mistralrs", features = ["metal"] } +mistralrs-core = { version = "0.9.5", package = "onde-mistralrs-core", features = ["metal"] } # Windows: CPU-only baseline (no GPU features). # For NVIDIA GPU builds, patch this to features = ["cuda"] in CI before building. [target.'cfg(target_os = "windows")'.dependencies] -mistralrs = { version = "0.9.3", package = "onde-mistralrs" } +mistralrs = { version = "0.9.5", package = "onde-mistralrs" } [dependencies] anyhow = "1" @@ -119,14 +119,16 @@ dotenvy = "0.15" uniffi = { workspace = true, features = ["build"] } # The target-conditional mistralrs deps above resolve to the published -# onde-mistralrs{,-core} 0.9.3 crates (setoelkahfi/mistral.rs fork, -# fix/all-platform-fixes, synced with upstream 0.9.x). Those in turn pull +# onde-mistralrs{,-core} 0.9.5 crates (setoelkahfi/mistral.rs fork, +# fix/all-platform-fixes, synced with upstream 0.9.5). Those in turn pull # onde-candle-{core,nn} 0.11.0 (candle rev 27f20fea published under new names; # upstream mistral.rs needs post-0.11.0 candle APIs no released candle has). -# For local dev against the fork, point them back at -# git = "https://github.com/setoelkahfi/mistral.rs", branch = "fix/all-platform-fixes" -# with package = "onde-mistralrs{,-core}" (the fork's crates carry the onde names -# now); flip to registry form before publishing, cargo publish rejects git deps. +# To test unpublished fork changes without editing this file, patch by the real +# package name from the command line (a [patch.crates-io] rename is silently +# ignored, but a patch keyed on the onde-* name works): +# cargo check --config 'patch.crates-io.onde-mistralrs.git="https://github.com/setoelkahfi/mistral.rs"' \ +# --config 'patch.crates-io.onde-mistralrs.branch="fix/all-platform-fixes"' +# and the same for onde-mistralrs-core. Do not commit the Cargo.lock that results. # Patch sysctl with watchOS support until upstream merges it. # PR: https://github.com/johalun/sysctl-rs/pull/74 diff --git a/sdk/dart/CHANGELOG.md b/sdk/dart/CHANGELOG.md index 6ac3594..4d367f0 100644 --- a/sdk/dart/CHANGELOG.md +++ b/sdk/dart/CHANGELOG.md @@ -1,3 +1,8 @@ +## 1.3.4 + +- Version aligned with the Rust core. No Dart API changes. +- The core now builds on `onde-mistralrs` `0.9.5`. Paged attention is turned off on iOS, tvOS, visionOS and watchOS to stay inside the memory budget there. + ## 1.3.3 - Version aligned with the Rust core. No Dart API changes. diff --git a/sdk/dart/pubspec.yaml b/sdk/dart/pubspec.yaml index b6b8d82..0125cf8 100644 --- a/sdk/dart/pubspec.yaml +++ b/sdk/dart/pubspec.yaml @@ -2,7 +2,7 @@ name: onde_inference description: >- On-device LLM inference for Flutter & Dart. Run Qwen 2.5 models locally with Metal on iOS and macOS, CPU on Android and desktop. No cloud, no API key. -version: 1.3.3 +version: 1.3.4 repository: https://github.com/ondeinference/onde homepage: https://ondeinference.com documentation: https://ondeinference.com/sdk/dart diff --git a/sdk/dart/rust/Cargo.lock b/sdk/dart/rust/Cargo.lock index 9b27ac5..9b2df99 100644 --- a/sdk/dart/rust/Cargo.lock +++ b/sdk/dart/rust/Cargo.lock @@ -3604,7 +3604,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" [[package]] name = "onde" -version = "1.3.3" +version = "1.3.4" dependencies = [ "anyhow", "cc", diff --git a/sdk/kotlin/gradle.properties b/sdk/kotlin/gradle.properties index 0e4d26b..ed56f80 100644 --- a/sdk/kotlin/gradle.properties +++ b/sdk/kotlin/gradle.properties @@ -10,7 +10,7 @@ kotlin.code.style=official # Maven publishing coordinates — keep in sync with Cargo.toml [package] version GROUP=com.ondeinference POM_ARTIFACT_ID=onde-inference -VERSION_NAME=1.3.3 +VERSION_NAME=1.3.4 diff --git a/sdk/react-native/CHANGELOG.md b/sdk/react-native/CHANGELOG.md index b229132..d7931bb 100644 --- a/sdk/react-native/CHANGELOG.md +++ b/sdk/react-native/CHANGELOG.md @@ -1,3 +1,8 @@ +## 1.3.4 + +- Version aligned with the Rust core. No React Native API changes. +- The core now builds on `onde-mistralrs` `0.9.5`. Paged attention is turned off on iOS, tvOS, visionOS and watchOS to stay inside the memory budget there. + ## 1.3.3 - Version aligned with the Rust core. No React Native API changes. diff --git a/sdk/react-native/package-lock.json b/sdk/react-native/package-lock.json index c91185c..8631279 100644 --- a/sdk/react-native/package-lock.json +++ b/sdk/react-native/package-lock.json @@ -1,12 +1,12 @@ { "name": "@ondeinference/react-native", - "version": "1.3.3", + "version": "1.3.4", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@ondeinference/react-native", - "version": "1.3.3", + "version": "1.3.4", "license": "MIT OR Apache-2.0", "devDependencies": { "expo-modules-core": "^2.0.0", diff --git a/sdk/react-native/package.json b/sdk/react-native/package.json index 76bc203..7e1c00d 100644 --- a/sdk/react-native/package.json +++ b/sdk/react-native/package.json @@ -1,6 +1,6 @@ { "name": "@ondeinference/react-native", - "version": "1.3.3", + "version": "1.3.4", "description": "On-device LLM inference for React Native. Run Qwen 2.5 models locally with Metal on iOS, CPU on Android. No cloud, no API key.", "main": "build/index.js", "types": "build/index.d.ts", diff --git a/sdk/react-native/rust/Cargo.lock b/sdk/react-native/rust/Cargo.lock index 06780f1..999eaa7 100644 --- a/sdk/react-native/rust/Cargo.lock +++ b/sdk/react-native/rust/Cargo.lock @@ -3420,7 +3420,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" [[package]] name = "onde" -version = "1.3.3" +version = "1.3.4" dependencies = [ "anyhow", "cc", @@ -3777,7 +3777,7 @@ dependencies = [ [[package]] name = "onde-react-native" -version = "1.3.3" +version = "1.3.4" dependencies = [ "jni", "log", diff --git a/sdk/react-native/rust/Cargo.toml b/sdk/react-native/rust/Cargo.toml index f959987..04c786c 100644 --- a/sdk/react-native/rust/Cargo.toml +++ b/sdk/react-native/rust/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "onde-react-native" -version = "1.3.3" +version = "1.3.4" edition = "2021" publish = false diff --git a/src/hf_cache.rs b/src/hf_cache.rs index fa71ef3..edb703e 100644 --- a/src/hf_cache.rs +++ b/src/hf_cache.rs @@ -1084,6 +1084,18 @@ where builder = builder.with_chat_template(chat_template); } + // Upstream #2493 made Metal paged attention the default; skip it on mobile + // so the validation load does not reserve the KV block pool. + #[cfg(any( + target_os = "ios", + target_os = "tvos", + target_os = "visionos", + target_os = "watchos" + ))] + { + builder = builder.without_paged_attn(); + } + // Shared flag so the monitor task knows when to stop. let finished = Arc::new(AtomicBool::new(false)); let finished_clone = Arc::clone(&finished); diff --git a/src/inference/engine.rs b/src/inference/engine.rs index 5299e32..e82c554 100644 --- a/src/inference/engine.rs +++ b/src/inference/engine.rs @@ -472,17 +472,23 @@ impl ChatEngine { // so the auto device mapper allocates far less memory. #[cfg(target_os = "tvos")] { - use mistralrs::DeviceMapSetting; - use mistralrs_core::AutoDeviceMapParams; - builder = builder .with_max_num_seqs(1) .with_prefix_cache_n(None) .with_no_kv_cache() - .with_device_mapping(DeviceMapSetting::Auto(AutoDeviceMapParams::Text { - max_seq_len: 512, - max_batch_size: 1, - })); + .with_max_model_len(512); + } + + // Upstream #2493 made Metal paged attention the default; its KV block + // pool is too large for the memory budget of Apple's mobile targets. + #[cfg(any( + target_os = "ios", + target_os = "tvos", + target_os = "visionos", + target_os = "watchos" + ))] + { + builder = builder.without_paged_attn(); } // On Android the GGUF embedded tokenizer is not supported by the @@ -638,17 +644,23 @@ impl ChatEngine { #[cfg(target_os = "tvos")] { - use mistralrs::DeviceMapSetting; - use mistralrs_core::AutoDeviceMapParams; - builder = builder .with_max_num_seqs(1) .with_prefix_cache_n(None) .with_no_kv_cache() - .with_device_mapping(DeviceMapSetting::Auto(AutoDeviceMapParams::Text { - max_seq_len: 512, - max_batch_size: 1, - })); + .with_max_model_len(512); + } + + // Upstream #2493 made Metal paged attention the default; its KV block + // pool is too large for the memory budget of Apple's mobile targets. + #[cfg(any( + target_os = "ios", + target_os = "tvos", + target_os = "visionos", + target_os = "watchos" + ))] + { + builder = builder.without_paged_attn(); } if let Some(ref template) = config.chat_template {