pwa(ai): turn WebGPU/worker local inference into a zero-install offline AI differentiator

[qnbs]

Parent

Part of #478 — PWA first-class excellence/hardening.

Why this is a genuine PWA differentiator

WorldScript already has a layered local-AI architecture, including WebLLM worker-offload ADRs, ONNX/WASM/Transformers-related paths and an existing local-AI facade. The browser can turn this into something native packaging does not automatically provide:

open/install the PWA, download a compatible model once, then use selected AI features locally/offline with WebGPU or WASM — without installing a separate WorldScript runtime.

This issue does not propose replacing Ollama or cloud providers. It makes browser-native inference a polished first-class execution target.

Existing architectural foundation

Relevant repository artifacts already include:

• docs/adr/0002-local-ai-stack-layering.md;
• docs/adr/0005-webllm-worker-offload.md;
• docs/LOCAL-AI.md;
• services/localAiFacade.ts;
• local-AI settings/UI and AI-core abstractions.

Therefore implementation should harden and productize the existing layering, not create another parallel AI stack.

Product modes

Expose execution truth clearly:

Browser local — WebGPU
Browser local — WASM/CPU
Local server — Ollama / compatible endpoint
Cloud provider
Unavailable / fallback

Never label a path “local” if manuscript context leaves the device.

Capability negotiation

Build one typed capability report covering:

• WebGPU availability;
• adapter/device acquisition;
• relevant limits/features;
• worker support;
• WASM/SIMD/threads where architecture uses them;
• available memory/storage estimate where exposed and safe;
• model compatibility;
• cross-origin-isolation requirements for any optional accelerated path;
• battery/mobile constraints where relevant.

Do not infer “WebGPU exists = model will work.” Capability admission must be model/workload specific.

Worker-first execution

Heavy inference/model initialization must remain off the editor/UI thread.

Goals:

• dedicated worker ownership;
• typed request/stream/cancel protocol;
• cancellation and timeout;
• crash/device-lost recovery;
• bounded queues/backpressure;
• no unbounded prompt/result buffering;
• editor/autosave responsiveness remains prioritized.

Coordinate cross-tab model ownership with #480 so two windows do not load duplicate GPU copies unnecessarily.

Model lifecycle manager

Create a coherent model lifecycle rather than treating browser caches as opaque:

available remotely
→ download requested
→ downloading
→ integrity verified
→ ready
→ loading
→ active
→ idle/unloaded
→ update available
→ removable

Users should see:

• model name/version/quantization where known;
• approximate download and stored size;
• execution backend;
• download progress;
• whether model is usable offline;
• last error/recovery action;
• remove/redownload controls.

Coordinate large artifact storage with #482 rather than abusing service-worker precache.

Integrity / supply-chain

For downloaded model artifacts:

• use pinned/versioned manifests;
• verify expected size/hash where distribution architecture permits;
• fail closed on corrupted/incomplete artifacts;
• do not execute/load a partially downloaded model as if complete;
• distinguish trusted project-owned model metadata from arbitrary user-provided model URLs;
• document upstream model/license provenance.

Progressive download / storage

Where model format/runtime permits, evaluate streaming/chunked download and OPFS-backed artifacts. Do not require entire multi-GB blobs to coexist multiple times in JS heap.

Measure before adopting complexity.

Device-lost and memory-pressure behavior

WebGPU device loss must be a normal recoverable state, not a white-screen failure.

On device loss/OOM/model-load failure:

1. cancel/settle affected requests;
2. release stale resources;
3. preserve manuscript/autosave state;
4. offer retry with same backend;
5. optionally fall back to smaller model/WASM where policy allows;
6. never silently switch to cloud.

Smart backend selection

Optional “Auto” mode may select among compatible local backends based on measured capability, but it must remain explainable.

Examples:

• WebGPU fast path;
• WASM fallback for smaller task models;
• Ollama when explicitly configured/selected;
• cloud only with explicit provider/user policy.

No hidden local→cloud escalation.

Task-specific local AI

Prioritize smaller, high-value models/tasks before assuming full LLM generation is always the best browser workload:

• embeddings/RAG;
• semantic similarity;
• classification/tagging;
• lightweight grammar/style helpers;
• entity extraction;
• local summarization where model permits;
• optional generative writing with supported WebLLM models.

This can make many daily writing-assistant operations instant/private even when a larger cloud/Ollama model remains available for generation.

Offline contract

After required artifacts are downloaded, explicitly test:

• cold offline PWA launch;
• model load offline;
• local inference offline;
• project edit/autosave during inference;
• app/service-worker update interactions;
• model cache survives normal app updates;
• stale/incompatible model version produces actionable UX.

Performance evidence

Track separately:

• model download throughput;
• cold model load;
• warm model load;
• time to first token/result;
• tokens/s or task throughput;
• UI input p50/p95 while inference runs;
• autosave latency during inference;
• GPU/JS memory where observable;
• cancellation latency;
• device-lost recovery.

A faster model that makes the editor janky is not a successful PWA optimization.

Privacy

• no prompt/manuscript telemetry;
• no hidden cloud fallback;
• explain which backend is processing current content;
• model download requests must not contain manuscript data;
• diagnostic logs redact prompt/output content by default.

Browser matrix

Chromium/WebGPU may be the enhanced tier. Other browsers may use WASM or report capability limitations. Basic writing must remain independent of local-AI availability.

Acceptance criteria

• Existing local-AI architecture remains the single product abstraction.
• Typed browser-local capability/admission report exists.
• Heavy inference runs off the UI thread.
• Cancellation/device-loss/OOM are recoverable.
• Model download/storage lifecycle is visible and manageable.
• Downloaded artifacts have integrity/version policy.
• Offline-after-download works for admitted models/tasks.
• No hidden local→cloud fallback exists.
• Multi-tab duplicate model initialization is prevented or bounded (pwa(concurrency): make multi-tab editing safe with Web Locks, BroadcastChannel & lifecycle ownership #480).
• Large model storage coordinates with pwa(storage): engineer a durable IDB+OPFS storage tier with persistence, quota & recovery UX #482.
• Editor/autosave responsiveness has explicit budgets during inference.
• Browser support/fallback is truthful.

Non-goals

• promising every LLM runs well in every browser;
• replacing Ollama;
• downloading large models without user intent;
• precaching models in the Service Worker app shell;
• sacrificing editor responsiveness for benchmark throughput;
• silently sending local tasks to cloud.

[qnbs]

Architecture refinement / single-AI-stack enforcement — 2026-08-24

This issue already has the correct direction: productize the existing local-AI layering; do not create another parallel AI stack. Make that a hard implementation constraint.

Existing authority to extend

• services/localAiFacade.ts remains the product-facing local inference abstraction;
• AI-core remains shared AI execution/domain infrastructure;
• WorkerBus v2 remains the heavy worker/task path;
• adaptiveAiEngine.ts remains backend/task selection infrastructure;
• localAiDeviceProfiler.ts / WebGPU detector remain capability seeds;
• existing GPU resource management, progress emitters, cancellation and model-ready state remain reusable lifecycle primitives;
• existing tab-leader logic is the current multi-tab seed until pwa(concurrency): make multi-tab editing safe with Web Locks, BroadcastChannel & lifecycle ownership #480 deliberately composes/replaces it.

Do not add PwaAiService, a second backend selector, a second worker queue, or a duplicate WebGPU capability detector.

Legitimate new component

A durable model-artifact manifest/catalog is justified only if current code lacks authoritative lifecycle facts such as:

• model/version/quantization identity;
• expected size/hash/integrity status;
• stored generation/location;
• compatible backend/runtime contract;
• download/ready/active/removable state;
• provenance/license metadata.

If introduced, it belongs beneath the existing local-AI facade and coordinates storage through #482.

Core boundary

Do not port browser WebGPU/WebLLM implementation to Rust merely to make PWA and Qt symmetrical. Portable request/result/error/cancellation/provider-policy semantics may move toward Core when justified; browser WebGPU remains a PWA execution adapter.

Acceptance additions

• Existing local-AI facade remains the single product abstraction.
• pwa(capabilities): establish progressive-enhancement registry, browser support tiers & runtime diagnostics #489 aggregates runtime capability; AI owns model/workload admission.
• pwa(concurrency): make multi-tab editing safe with Web Locks, BroadcastChannel & lifecycle ownership #480 coordination does not leave a second permanent AI leader authority.
• pwa(storage): engineer a durable IDB+OPFS storage tier with persistence, quota & recovery UX #482 owns large-artifact storage policy rather than service-worker precache.
• Any new model catalog has one authoritative lifecycle state and migration/version policy.
• No hidden local→cloud escalation exists under error/fallback pressure.