[iChubai]

Summary

• Execute all twelve WorldMemBench metric protocols in tree, with official per-model result import and scorecards.
• Reuse canonical CLIP, LAION heads and DINO loading. Human/object appearance share one masked extractor; compatible DINO checkpoint/device pairs share a backbone while retaining independent preprocessing and batching.
• Share WorldScore/4DWorldBench IQA preprocessing and reuse existing LAION heads in MIND and WorldScore. The WorldScore aesthetic metric does not require Lightning.
• Bind WBench imaging quality explicitly to its official MUSIQ-KonIQ checkpoint without changing PyIQA's global cache. WorldMemBench retains MUSIQ-SPAQ and its distinct preprocessing.

Scope

• Benchmarks: WorldMemBench, WorldScore, 4DWorldBench, MIND and WBench.
• Entrypoints: worldfoundry-eval zoo benchmark-run --benchmark-id worldmembench and worldfoundry.evaluation.tasks.execution.runners.worldmembench.run_worldmembench_official_runner.
• Outputs: official per-model summaries plus scorecard.json, raw_metric_table.jsonl and per_sample_scores.jsonl. Units, directions, coverage and failures retain their official meaning; no cross-metric composite is introduced.
• Shared implementations own model loading and equivalent preprocessing. Benchmark-specific sampling, revisits, alignment, VLM judging and aggregation remain separate.
• Compatible DINO consumers reuse model memory per run without sharing adaptive batch-size state. Metric assets and datasets remain outside the repository.

Runtime And Assets

• Validation uses CPU execution in Python 3.11, including the prepared unified environment's Torch 2.5.1+cu121 and torchvision 0.20.1+cu121. No CUDA kernels or pretrained checkpoints are needed for these tests.
• WorldMemBench source: https://github.com/ZJU-REAL/WorldMemBench at 1b79840143a98e017891e893b5d3ef75b93078a3 (MIT). Dataset: rescuerz/WorldMemBench.
• Selected metric dependencies are listed in worldfoundry/data/benchmarks/requirements/worldmembench.txt. Use the prepared GPU environment without replacing its Torch/CUDA ABI.
• --weights-dir accepts the official asset layout; YAML checkpoint overrides can reuse existing staged assets.
• DA3-Streaming and full-profile ViPE preparation accept staged artifacts or configured model runtimes. Dynamic Memory uses the configured VLM endpoint and API-key environment variable.

Validation

139 scoped tests pass across the fast CPU and unified-environment selections:

# Fast CPU environment: 105 passed
PYTHONPATH="$PWD" OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m pytest -q \
  tests/eval_core/test_worldmembench_official_runner.py \
  tests/eval_core/test_worldmembench_runtime.py \
  tests/eval_core/test_eval_tasks_fix_official_runner.py \
  tests/eval_core/test_benchmark_hub_docs.py \
  tests/eval_core/test_runner_kind_catalog.py \
  tests/eval_core/test_wbench_official_runner.py \
  tests/evaluation/execution/test_helper_reuse.py \
  tests/evaluation/execution/test_worldscore_evaluator_lifecycle.py --tb=short

# Unified environment, CPU tensors and OpenCV, no pretrained assets: 13 passed
PYTHONPATH="$PWD" OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 "$PYTHON" -m pytest -q \
  tests/evaluation/execution/test_shared_quality.py \
  tests/evaluation/execution/test_worldmembench_camera.py --tb=short

# Existing quality regressions in the unified environment: 21 passed
PYTHONPATH="$PWD" OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 "$PYTHON" -m pytest -q \
  tests/evaluation/execution/test_shared_scoring_backend.py \
  tests/evaluation/execution/test_vbench_appearance_reduction.py --tb=short

Coverage includes exact CPU equality for LAION head predictions and IQA transforms, explicit DINO CLS/pooler selection, backbone sharing, masked single/batch extraction, MUSIQ variant selection and OpenCV resizing, stride-four sampling, equal-video aggregation, event gating, native camera replay from synthetic poses, standard CLI dispatch and failure coverage.

Command	Result	Notes
python -m pre_commit run	Passed	Staged files, Ruff, ty and repository lint
make pr-check BASE_REF=origin/main	Passed	Committed Python checks and DCO
make data-check workspace-registry-check	Passed	Catalog and dispatch integrity
python scripts/setup/check_packaging_license_gate.py	Passed	Public package boundaries
bash scripts/docs/build.sh --skip-install	Passed	Complete English/Chinese static documentation build

Paired WorldMemBench and WBench pages describe checkpoint profiles, shared components, execution and validation status. The catalog declares in_tree_official_runtime; leaderboard eligibility remains false.

Known Limitations

Real-checkpoint GPU and complete-dataset replay remain pending. The synthetic camera replay and CPU numerical tests do not establish leaderboard parity. DA3-Streaming/full-profile ViPE generation requires its configured model runtime when artifacts are absent, and Dynamic Memory requires a provisioned VLM endpoint.

Contribution Requirements

• The PR addresses one focused change and includes regression tests.
• Public behavior and paired English/Chinese docs are updated.
• Upstream attribution and license records are retained.
• No credentials, checkpoints, caches, or generated run artifacts are committed.
• Every contribution commit has a DCO sign-off.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.