Perf migration

[anindex]

Runtime

• In-tree SigLIP vision towers for pi0/pi0.5/smolvla; fetch and pin llama.cpp b9866 via FetchContent (no patch).
• One shared gguf_reader across the model loaders (was duplicated per arch).
• Quantization: scripts/quantize_gguf.py repacks LM weights to Q8_0/Q4_0 and the loader runs quantized GGUFs directly. Q8_0 is near-lossless and roughly halves the LM.
• vla-cli: one-shot inference from an image plus tokens, no server or simulator.
• Reuse the reprojected prefix cross-attention K/V across denoise steps in smolvla; byte-identical output, about 5% off LM inference (383 to 364 ms on CPU over 30 iters).
• Scale CPU threads to the machine core count across all loaders instead of a fixed 4.

Fixes

• evo1 base noise N(0,1) and unmasked DiT cross-attention to match the reference.
• Out-of-range language-token guards (smolvla, openvla_oft, vla_adapter); padded action dims zeroed.
• Server bounds image size and payload with clean shutdown; precomputed_img_emb and null-image guards.
• Reject images that do not match the model input size in openvla_oft and vla_adapter (out-of-bounds read on a smaller view).
• Validate evo1 action dims at load (action_dim == horizon * per_action_dim) so a client noise buffer cannot underrun.
• Enable the bitvla CUDA LM only after every device buffer is allocated, else fall back to CPU.
• Cap inbound message size (256 MiB) and image count (16) on vla-server and vlm-server.
• Read gguf file offsets as 64-bit and reject non-float embedding tensors in row-fetch.

Infra

• Dockerfile (CPU and CUDA) and a GitHub Actions build gate, both building vla-server, vlm-server, and vla-cli.
• docs/ARCHITECTURE.md design overview and a README quickstart (download, merge mmproj, one vla-cli prediction); document attention_mask as evo1-only.

Validation

• Clean CPU and CUDA builds under -Wall -Wextra.
• smolvla libero_object 100%.
• gr00t Q8_0 vs bf16 on libero_object task_0: 100% (5/5) vs 80% (4/5), within variance.
• predict_check output byte-identical across 8 models for the shared-reader change, and for smolvla across the
  cross-attention change.

[khanhnd61-vr]

LIBERO sweep report - libero_object_sweep

• Sweep root: /home/khanh/work/vla.cpp/outputs/libero_object_sweep
• Suite: libero_object (tasks 0..9)
• Generated: 2026-07-06T17:41:06

Reproducibility

Generated by scripts/print_versions.sh on 2026-07-06T10:41:06Z.

vla.cpp

• repo HEAD: 1c22959ba97dd705bbd7b68f5ca2b108eab257ef (1c22959)

llama.cpp (third_party)

• HEAD: 75a48a90559abf65df3f3616a53bb16e5afb9d07 (b9866)
• expected pinned tag (from CMakeLists.txt): ?

GGUF checksums

(pass GGUF paths as positional args to include their sha256s)

Host

uname:   Linux 6.8.0-124-generic #124~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 26 21:05:19 UTC  x86_64 x86_64 x86_64 GNU/Linux
os:      Ubuntu 22.04.5 LTS
nvidia:  NVIDIA GeForce RTX 3090, 595.71.05
         24576 MiB, 8.6
nvcc:    Cuda compilation tools, release 12.8, V12.8.93
cmake:   cmake version 3.22.1
g++:     g++ (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
python3: Python 3.10.12
uv:      uv 0.11.19 (x86_64-unknown-linux-gnu)

Python venv pins

LIBERO venv (/home/khanh/work/vla.cpp/eval/sim/libero/libero_uv/.venv):

(venv not found at /home/khanh/work/vla.cpp/eval/sim/libero/libero_uv/.venv)

Determinism

• LIBERO seeds: see eval/client/run_libero_eval.py --seed (default in that script).
• Server-side denoising noise: client-supplied if PredictRequest.noise is set;
  otherwise sampled from the per-arch RNG seeded at model load (not currently
  exposed on the CLI - set PredictRequest.noise from the client for bit-exact
  reruns).

Success rate & client-side inference time

• SR counts terminated episodes as failures: successes / n_episodes.
• client/step - wall-time per env step (amortized over chunk replay; matches Average inference time per step in each summary.txt).
• client/call = client/step × n_action_steps - wall-time per actual vla-server call. Includes client pre/post + ZMQ transport (TCP loopback) + server compute.

Model	n_act	Tasks	Successes	Terminated	SR	client/step (ms)	client/call (ms)
bitvla	8	10	10/10	0/10	100.00%	18.16	145.26
evo1	8	10	9/10	0/10	90.00%	29.73	237.87
gr00t_n1_5	16	10	10/10	0/10	100.00%	6.80	108.85
gr00t_n1_6	16	10	8/10	0/10	80.00%	6.20	99.25
gr00t_n1_7	16	10	10/10	0/10	100.00%	6.36	101.70
openvla_oft	8	10	10/10	0/10	100.00%	32.06	256.44
pi0	50	10	8/10	0/10	80.00%	5.27	263.65
pi05	10	10	10/10	0/10	100.00%	16.66	166.56
smolvla	1	10	10/10	0/10	100.00%	85.66	85.66
vla_adapter	8	10	10/10	0/10	100.00%	13.49	107.92

Server-side inference breakdown

Parsed from _server_logs/<arch>.log lines:

vla-server: rid=…  served=…  total=… ms  vision=…  inf=…  other=…

These are server-side measurements only - they exclude ZMQ transport and client pre/post. total = vision + inf + other.

Model	Samples	total (ms)	vision	inf	other
bitvla	17	111.00	20.18	90.81	0.00
evo1	23	227.92	132.39	81.10	14.42
gr00t_n1_5	9	86.14	27.12	52.87	6.18
gr00t_n1_6	15	95.28	22.33	43.63	29.29
gr00t_n1_7	10	73.52	26.16	41.79	5.56
openvla_oft	16	230.02	45.08	183.98	0.94
pi0	4	123.47	25.85	91.68	5.95
pi05	19	137.76	25.56	105.13	7.08
smolvla	152	72.05	36.80	33.65	1.60
vla_adapter	18	84.23	43.54	40.54	0.18

Transport + client overhead

overhead = client/call − server total - time spent outside vla-server (ZMQ over loopback + client preprocessing + protobuf round-trip).

Model	client/call (ms)	server total (ms)	overhead (ms)
bitvla	145.26	111.00	34.26
evo1	237.87	227.92	9.95
gr00t_n1_5	108.85	86.14	22.70
gr00t_n1_6	99.25	95.28	3.97
gr00t_n1_7	101.70	73.52	28.18
openvla_oft	256.44	230.02	26.42
pi0	263.65	123.47	140.18
pi05	166.56	137.76	28.80
smolvla	85.66	72.05	13.62
vla_adapter	107.92	84.23	23.69

Peak memory

Sampled by the inline mem_sampler function in eval/run_libero.sh while vla-server was alive:

• Peak VRAM - max of per-PID used_memory from nvidia-smi --query-compute-apps, polled every 1s. (no GPU) on Tegra/Jetson, which doesn't support that query.
• Peak RAM - VmHWM from /proc/<pid>/status (kernel-tracked high-water mark of resident memory). Host only - does not include the iGPU's unified-memory allocations.
• Peak sys RAM / sys Δ - peak system-wide used RAM (MemTotal - MemAvailable) and its rise over the sampler-start baseline. On Tegra (unified memory) this is the only metric that captures the iGPU weights VRAM/VmHWM miss; the Δ is an upper bound on the server's footprint (a co-resident client/sim is included).

Model	Peak VRAM (MiB)	Peak RAM (MiB)	Peak sys RAM (MiB)	sys Δ (MiB)	Samples
bitvla	1464	1181.9	9483.2	3140.4	118
evo1	1716	736.6	9564.2	3436.5	135
gr00t_n1_5	5062	1439.3	6871.4	3079.9	122
gr00t_n1_6	6098	1443.6	6933.8	3284.5	96
gr00t_n1_7	6456	1419.8	6976.0	3280.5	109
openvla_oft	14866	1039.6	6853.9	682.3	208
pi0	5672	705.4	9716.2	3853.7	140
pi05	6128	726.7	9642.6	3024.6	203
smolvla	1588	766.4	9715.6	3526.4	213
vla_adapter	3086	1066.8	9839.7	3347.5	117

Client/call improvement vs ci/baselines/rtx3090.json

client/call (ms) from the table above vs the RTX 3090 baseline (Δ% negative = faster).

Model	baseline (ms)	report (ms)	Δ (ms)	Δ%
evo1	508.84	237.87	-270.97	-53.3%
bitvla	302.78	145.26	-157.52	-52.0%
gr00t_n1_5	226.69	108.85	-117.84	-52.0%
gr00t_n1_6	164.64	99.25	-65.39	-39.7%
gr00t_n1_7	164.16	101.70	-62.46	-38.0%
smolvla	112.63	85.66	-26.97	-23.9%
pi0	311.68	263.65	-48.03	-15.4%
vla_adapter	120.00	107.92	-12.08	-10.1%
openvla_oft	256.00	256.44	+0.44	+0.2%
pi05	161.00	166.56	+5.56	+3.5%
total	2328.42	1573.16	-755.26	-32.4%

Bottom line: 8 of 10 models are faster than baseline.