CLAMP: Constrained Decoding for Vision-Language Embodied Planning

Tianyi Ma  ·  Parisa Kordjamshidi
Michigan State University · Heterogeneous Learning and Reasoning (HLR) Lab

Website · Camera-ready manuscript · Code · Model checkpoints · Dataset & demos · BibTeX

CLAMP turns visual evidence and supplied symbolic rules into decoding-time constraints for grounded embodied planning. It keeps the vision-language model frozen: hard masks remove syntactically invalid, visually unsupported, or transition-infeasible continuations, while an optional HMM lookahead ranks the choices that remain.

A fluent plan is not necessarily an executable plan. CLAMP constrains the distribution before an invalid action enters the sequence.

Method

For each episode, CLAMP builds an observation-conditioned planning interface:

1. Observe: the image and task instruction are provided to the frozen VLM.
2. Compile: syntax and observation-conditioned visibility become a token-level DFA; the supplied symbolic action model becomes an action-level DFA for transitions and goal reachability.
3. Decode: the two hard recognizers reject invalid tokens and grounded actions; an optional HMM prior re-ranks only the feasible set.

CLAMP-standard returns FAIL or a plan accepted by the supplied interface, assuming sound perception, specification, and masks. It does not by itself validate those inputs or downstream physical execution.

Results

Evaluation	Camera-ready result	Evidence boundary
VLABench, frozen Qwen3-VL-8B, 480 prompts	Weighted-DSL: baseline 28.7, DFA only 34.1, text HMM 37.1, supervised vision HMM 38.7	The final row uses benchmark skill/image pairs plus per-instance calibration. With no matched no-calibration row, 37.1→38.7 is a cumulative configuration contrast—not an isolated TTA effect.
InternVL3.5-8B, matched four-dimension subset, 474 prompts	Entity-ID validity 0.019→1.000; format validity remains 1.000	Supports interface conformance, not a claim that CLAMP adds visual knowledge.
SafeAgentBench, legitimate long horizon	CLAMP-policy reduces symbolic violations 0.41→0.05 and raises safe completion 0.06→0.18	Uses backup-and-retry plus validated forced-action injection; this is not the one-pass CLAMP-standard decoder.
TaPA-60 grounding diagnostic	Out-of-scene references 10.0%→8.3%, while matched plan success 71.7%→68.3%	A grounding diagnostic, not a task-success gain; the self-judge is sensitive to harness metadata.
VH→BEHAVIOR HMM calibration	Conditional log-likelihood 7.132→3.095 (56.6%) in 417 seconds	VLM frozen; planning is mixed, with single-skill collapse on five dimensions.

These results measure plan-level validity and benchmark-specific outcomes. They should not be read as a universal guarantee of simulator or robot execution success.

Qualitative examples

Successful constrained plan and failed baseline

For the instruction “This dish lacks a deep, tangy touch. Maybe adding something could elevate its taste,” the unconstrained baseline emits pick("bbq_sauce") → pour("pan_seen") → place("dishes"), which is schema invalid and scores 0.222. CLAMP emits the grounded identifier plan pick(4) → pour(2), matching the reference and scoring 1.000.

This InternVL3.5 measured trace uses the non-interactive VLABench plan evaluator. The DFA supplies schema compliance, visibility supplies entity support, and the HMM raises STOP probability from 11.91% to 69.83%. The case demonstrates interface compliance and correct termination—not improved semantic recognition, a main Qwen3-VL result, or simulator execution success.

Plan-match example	Counterexample
The predicted skill sequence and grounded target agree with the reference (plan-match score 80). This label does not imply simulator success.	The skill is plausible, but the plan selects the wrong flower entity (plan-match score 60); downstream execution also fails.

The public demonstration release contains 20 recorded VLABench episodes, balanced between plan-match examples and counterexamples, with inputs, ground-truth plans, evaluation records, and synchronized execution views.

Resources

• Website: CLAMP project page
• Paper: CLAMP camera-ready manuscript
• Code: HLR/CLAMP
• Model: CLAMP HMM Checkpoints
• Dataset: CLAMP Sampled Continuations and VLABench Demos
• Release collection: HMM Checkpoints, Sampled Continuations, and Demos
• Website source: project-page/
• Author contact: matiany3@msu.edu

Citation

@inproceedings{ma2026clamp,
  title     = {{CLAMP}: Constrained Decoding for Vision-Language Embodied Planning},
  author    = {Ma, Tianyi and Kordjamshidi, Parisa},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  publisher = {Association for Computational Linguistics}
}

Code release

The repository constructs scene-grounded constraints, compiles them into DFA masks, optionally adds an HMM prior or test-time adaptation, and can pass plans to VLABench/OpenPI execution.

This is a code release, not an artifact archive. It includes the implementation, small prompt subsets, configurations, and tests. It does not include model weights, full benchmark data, simulator assets, raw trajectories, private experiment reports, private paper material, or credentials.

Start here

git clone https://github.com/HLR/CLAMP.git
cd CLAMP

# Select an appropriate PyTorch build for the machine first, then install CLAMP.
pip install -e .

# CPU-only checks: no model weights, GPU, JAX, or VLABench installation needed.
python tests/test_schema_and_rule_maker.py
python tests/behavior/test_behavior_wiring.py

# Optional dependencies.
pip install -e ".[dev]"     # pytest
pip install -e ".[ttt]"     # JAX / Equinox test-time training
pip install -e ".[server]"  # OpenAI-compatible online planner client

The operational guide is docs/benchmark_guide.md. It documents each benchmark's inputs, loaders, commands, evaluator boundary, and result locations.

Repository map

Path	Responsibility	Main callers
clamp/config.py	Dataclasses and YAML configuration loading.	Every command in bin.
clamp/data	Load VLABench prompts/ground truth and EAI BEHAVIOR/VirtualHome prompt bundles.	run_clamp.py, run_clamp_behavior.py.
clamp/perception	Turns prompt-side scene metadata into a SceneState.	VLABench offline generation.
clamp/rule_maker	Builds and validates a ConstraintSpec from a scene and instruction.	VLABench offline generation.
clamp/dfa	Compiles VLABench constraints and routes EAI task families to their domain DFA builders.	Constrained decoders and the BEHAVIOR runner.
clamp/hmm_prior and clamp/ttt	Optional HMM loading, gamma look-ahead, and offline/per-instance adaptation.	HMM ablations and TTT scripts.
clamp/decoder, clamp/inference, clamp/agents	HuggingFace/vLLM constrained decoding and two-agent orchestration.	run_clamp.py, run_clamp_vllm.py.
clamp/execution	Episode bookkeeping, SkillLib bridge, and OpenPI bridge.	run_clamp_vlabench.py.
clamp/vendor	Attributed snapshots needed for DFA/Ctrl-G, conditional HMM, and VLABench integration.	Loaded lazily by CLAMP; see notices below.
bin	CLI entry points.	Run directly from the repository root.
configs	Default, ablation, and serving YAML files.	CLI defaults and overrides.
data	Versioned small BEHAVIOR and VirtualHome prompt subsets.	Smoke runs and schema checks.
project-page	Static research webpage, paper PDF, citation, figures, and qualitative evidence.	Packaged by scripts/publish_project_page.sh.
scripts	Benchmark wrappers, input preparation, TTT training, and demo utilities.	Run from the repository root.
tests	Dependency-light smoke tests plus TTT unit tests.	Development/CI.

Data flow and command boundaries

VLABench prompt JSON
  -> clamp/data/prompts.py
  -> perception + rule_maker
  -> ConstraintSpec -> clamp/dfa
  -> constrained decoder (+ optional HMM / TTT)
  -> offline plan/result JSON

BEHAVIOR or VirtualHome prompt JSON
  -> clamp/data/behavior_prompts.py
  -> BehaviorPromptBundle
  -> domain-specific DFA from clamp/dfa/behavior_builder.py
  -> baseline, gamma Ctrl-G, or two-level Ctrl-G output JSON

Identifier-keyed VLABench plan JSON
  -> bin/run_clamp_vlabench.py
  -> OpenPiSkillBridge -> VLABench MuJoCo
  -> metrics.json + per-task detail_info.json

The two VLABench paths deliberately have different roles:

• bin/run_clamp.py is the HuggingFace offline-generation runner. It writes a JSON list with generated plans and diagnostics.
• bin/run_clamp_planner_online.py writes an identifier-keyed plan object designed for bin/run_clamp_vlabench.py --plan-file.
• bin/run_clamp_vlabench.py is the live MuJoCo/OpenPI executor and produces success, intention, and progress metrics.

In particular, selecting the clamp_full ablation in the offline runner does not itself create a VLABench environment. Use the executor entry point for simulator metrics.

Command index

Entry point	Purpose	Typical output
bin/run_clamp.py	HuggingFace VLABench offline generation and the five main ablations.	A JSON list of plans/results.
bin/run_clamp_behavior.py	BEHAVIOR and VirtualHome baseline, gamma Ctrl-G, and two-level Ctrl-G generation.	A metadata-wrapped response JSON.
bin/run_clamp_planner_online.py	Call an OpenAI-compatible endpoint and build an identifier-keyed VLABench plan map.	Plan JSON consumable by the executor.
bin/run_clamp_planner_online_beta.py	Refine whole-task VLABench instructions against EAI entity lists.	Identifier-keyed task-plan JSON.
bin/run_clamp_vlabench.py	Execute a plan map or trivial baseline in VLABench through OpenPI.	metrics.json plus per-task details.
bin/run_clamp_vllm.py	Local vLLM two-agent/orchestrated generation path.	Result JSON and optional trajectories.
bin/serve.py	Launch a vLLM OpenAI-compatible API server.	A running inference service.
bin/train_ttt_gradient.py and bin/train_ttt_baum_welch.py	Low-level TTT training commands; wrappers set their usual defaults.	Adapted HMM checkpoints.

Benchmark entry points

Benchmark/workflow	Input loader and input	Run entry point	Generated artifacts
VLABench offline plan generation	clamp/data/prompts.py; external prompt JSON and optional ground-truth JSON.	bin/run_clamp.py or scripts/vlabench.	outputs/clamp/.json, logs/clamp/.log.
VLABench online planning + execution	Online planner reads the same prompt JSON; executor loads a plan map plus VLABench scenes.	bin/run_clamp_planner_online.py then bin/run_clamp_vlabench.py.	<save-dir>/metrics.json and <save-dir>/<task>/detail_info.json.
EAI BEHAVIOR	clamp/data/behavior_prompts.py; bundled subsets or EAI-generated HELM-style prompts.	bin/run_clamp_behavior.py and scripts/behavior.	outputs/behavior/<task>_<mode>.json, logs beside it.
EAI VirtualHome	The same BEHAVIOR loader, with virtualhome_* families and bundled subsets.	run_clamp_behavior.py with an explicit prompt file.	User-chosen output file, conventionally outputs/virtualhome/.

The public release intentionally excludes full VLABench prompts/scenes and the official EAI evaluator. The benchmark guide explains how CLAMP connects to those separately installed dependencies rather than silently relying on a particular local workspace.

Results, logs, and reproducibility

Generated artifacts are ignored by Git:

• Offline VLABench plans/results: outputs/clamp/; logs: logs/clamp/.
• BEHAVIOR generation results: outputs/behavior/; logs: logs/behavior/.
• VLABench OpenPI execution: the explicit --save-dir supplied to run_clamp_vlabench.py, containing metrics.json and per-task detail_info.json.
• TTT checkpoints: outputs/clamp/ttt_gradient/ or outputs/clamp/ttt_baum_welch/.

No claimed benchmark numbers or raw execution artifacts are bundled in this public repository. This separation keeps benchmark licenses, large simulator assets, and run-specific data out of source control.

Source context and acknowledgements

CLAMP was extracted from the private Control-Thinking-In-Embodied-Agents research workspace. The released implementation adapts components from eai_ctrlg, eai_train, and VLABench; EAI prompt/evaluation formats come from Embodied Agent Interface.

The shipped third-party snapshots, changes, and applicable upstream terms are listed in THIRD_PARTY_NOTICES.md. CaP-Demo, LLM-TAMP, and ltl_safety are project-context/baseline sources in the parent research workspace, organized through EmbodiedAgents-Code; their code is not redistributed in this repository.

License and use

This public release intentionally does not apply a blanket license to CLAMP-specific contributions. Vendored components retain their upstream terms, documented in THIRD_PARTY_NOTICES.md. Please contact the maintainers before redistributing CLAMP-specific code or data.