CamVR — Webcam Body Tracking for SteamVR / VRChat

Русская версия | English

Low-overhead body tracking from a single 1080p webcam (30-60 FPS) for SteamVR and VRChat. Tracks waist + knees + ankles (5 points) via MediaPipe and exposes them as SteamVR GenericTracker devices through a native OpenVR driver. Optimized to leave GPU/CPU headroom for PCVR.

Current version: v0.3 — 5 trackers, Z-inversion fix, mirror fix, HMD-anchored calibration. See Releases.

---

Features

• Single webcam 1080p, 30 FPS (60 FPS supported) → 5 SteamVR trackers: waist, left_knee, right_knee, left_ankle, right_ankle (configurable, config/config.yaml:21)
• HMD-anchored calibration: T-pose (c) computes scale + HMD offset via OpenVR (tracker/app.py:342) so trackers appear under the headset, not 2 m away at the camera
• OneEuro filter (tracker/utils/one_euro.py:1) — jitter-free without lag
• Mirror & Z fixes: flip_horizontal (config.yaml:11) and flip_z (config.yaml:35) with runtime toggle m/z (tracker/app.py:374)
• Lightweight: 640x360 inference, ~13 ms on i5-13600KF XNNPACK CPU (tracker/bench.py:1), ~4-6% CPU, 0% GPU (CPU mode) or ~5% GPU on RTX 5070
• Prebuilt driver: driver/dist/bin/win64/driver_camvr.dll (MinGW, 4.3 MB) + vrpathreg auto-install, or build from source (openvr + nlohmann/json)

Architecture

[1080p Webcam] --1280x720--> [OpenCV 640x360] --> [MediaPipe Pose Full] --> [Pose Mapping + HMD Offset]
    --> [OneEuro] --> [UDP 127.0.0.1:6969] --> [OpenVR Driver C++ 5x GenericTracker] --> [SteamVR / VRChat FBT]

• Capture 1280x720 → resize 640x360 for inference (4× saving)
• MediaPipe Pose model_complexity=1 (full) ~13 ms, Lite (0) ~8 ms if CPU-bound
• UDP JSON bridge (tracker/utils/ipc.py:1) — driver-agnostic, tools/fake_driver.py for testing without SteamVR

Hardware

• Tested on: i5-13600KF + RTX 5070, 1080p webcam, Oculus Quest 3 via Link (SteamVR oculus driver). Any 1080p 30 FPS camera works.
• Camera placement: tripod, ~2 m distance, ~1.2 m height, full body in frame. Desk-mounted crops legs.

Quick Start (Windows, PowerShell)

1. Environment — avoid Cyrillic path bug

MediaPipe's C++ binarypb fails on C:\Users\<Cyrillic>\.... Use a venv on D:\ (already configured as .venv):

python -m venv .venv
.venv\Scripts\pip.exe install -r requirements.txt
:: behind firewall/proxy:
.venv\Scripts\pip.exe install -r requirements.txt --proxy socks5h://127.0.0.1:10808
:: optional Lite/Heavy models:
.venv\Scripts\python.exe tools/download_models.py --proxy socks5h://127.0.0.1:10808

2. Check camera & benchmark

.venv\Scripts\python.exe tools/test_camera.py
.venv\Scripts\python.exe tracker/bench.py --frames 100 --complexity 1

Expected: Avg ~13 ms → OK for 60 FPS (33 ms budget for 30 FPS). If >33 ms, lower inference_width to 512 in config/config.yaml:9.

3. Preview without SteamVR

run.bat
:: or
.venv\Scripts\python.exe -m tracker --no-udp

Window CamVR - Body Tracking POC:

• Green/orange dots: waist/knees/ankles (tracker/utils/pose.py:44)
• Overlay: FPS / Infer / Scale / Mirror / Z
• Keys: q exit, c T-pose calibrate (scale + HMD offset), m mirror toggle, z flip Z, f filter toggle, r reset, d debug, h HMD pose

4. UDP test without driver

:: terminal 1
.venv\Scripts\python.exe tools/fake_driver.py
:: terminal 2
.venv\Scripts\python.exe -m tracker --headless --frames 100

Should print waist=+0.01,+0.95,+0.10 every 0.5 s.

5. SteamVR (prebuilt, no build needed)

Driver is prebuilt at driver/dist/ and driver/camvr_driver_win64.zip (v0.3), already copied to C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers\camvr and registered via vrpathreg.

After pulling:

:: if steamvr path differs, run:
powershell -ExecutionPolicy Bypass -File driver/install.ps1

Restart SteamVR → Settings → Manage Trackers shows 5 CAMVR devices (green when tracking). If grey, see Troubleshooting.

Build from source (optional, needs CMake 3.30 + MinGW/VS2022 + OpenVR SDK):

git clone https://github.com/ValveSoftware/openvr.git D:/openvr
mkdir driver\third_party\nlohmann -Force
curl.exe -L https://github.com/nlohmann/json/releases/download/v3.11.3/json.hpp -o driver\third_party\nlohmann\json.hpp
:: with proxy: curl.exe -x socks5h://127.0.0.1:10808 -L ...
build_driver.bat
:: or manually:
tools\cmake\bin\cmake.exe -S driver -B driver/build -G "MinGW Makefiles" -DOPENVR_ROOT="D:/openvr" -DCMAKE_MAKE_PROGRAM="C:/Program Files/TDM-GCC-64/bin/mingw32-make.exe"
tools\cmake\bin\cmake.exe --build driver/build --config Release
powershell -ExecutionPolicy Bypass -File driver/install.ps1

Calibration & VRChat

1. Start SteamVR (headset tracking), then run.bat. Log should show [HMD] SteamVR HMD pos [x y z] (tracker/app.py:178), not not found.
2. Stand T-pose 2 m from camera, face camera, c in tracker window. Console: [Calib] HMD [...] raw_waist [...] -> offset [...] (tracker/app.py:350). Filters reset.
3. SteamVR → Manage Trackers assign roles:
   • CAMVR_waist → WAIST (ПОЯС)
   • CAMVR_left_knee → LEFT KNEE (ЛЕВОЕ КОЛЕНО)
   • CAMVR_right_knee → RIGHT KNEE
   • CAMVR_left_ankle → LEFT FOOT (ЛЕВАЯ НОГА = foot, not ankle) for best VRChat leg IK, or LEFT ANKLE (ЛЕВАЯ ЛОДЫЖКА)
   • CAMVR_right_ankle → RIGHT FOOT (ПРАВАЯ НОГА)

   НОГА in SteamVR = foot (on top of foot). Choose НОГА for VRChat, ЛОДЫЖКА is ankle ~10 cm above.

4. VRChat: use an FBT-compatible avatar (not default robot). Search worlds Avatar Search for FBT/Full Body. Settings → Tracking & IK → Allow Full Body. Then Quick Menu → Calibrate FBT in T-pose. Check mirror — waist/knees/ankles should bend legs. If left moves right, press m and c again.

Keys: m mirror left/right, z flip Z (back↔forward), o show offset.

Config

config/config.yaml:

camera:
  fps: 30
  inference_width: 640
  flip_horizontal: true  # mirror fix for front camera
mediapipe:
  model_complexity: 1  # 0=lite, 1=full, 2=heavy
tracking:
  points: ["waist","left_knee","right_knee","left_ankle","right_ankle"]
calibration:
  use_hmd: true
  hmd_offset: [0,0,0]  # auto after c
  flip_z: false

Performance (VRChat)

Metric	Value	Note
Infer 640x360	~13 ms CPU	30 FPS budget 33 ms
CPU	4-6% i5-13600KF	pin to E-cores if needed
GPU	0% (CPU) or 5% RTX 5070	use_gpu: false → 0%
RAM	~300 MB
Latency	40-50 ms	camera 33 ms + infer 13 ms

If VRChat stutters: model_complexity:0 + 512x288 + fps:30.

Troubleshooting

• Grey hexagons in SteamVR: not tracking → check C:\tmp\camvr.log (driver/src/driver.cpp:26 logs every 90 pkts), firewall UDP 6969, camera full body, c calibrate with HMD. tools/check_udp.py should show packets.
• Left ↔ Right inverted: m toggle or flip_horizontal: true/false.
• Back ↔ Forward inverted: z toggle or flip_z: true.
• Trackers in floor / 2 m away: press c in T-pose with SteamVR running to compute HMD offset; ensure use_hmd: true.
• Avatar doesn't move but calibration dots move: use FBT avatar, enable Allow Full Body, VR not Desktop, and ensure roles are WAIST/FOOT.
• Cyrillic path binarypb fail: use .venv on D:\ (run.bat does this). Or tools/download_models.py with proxy for 403.
• Driver not loading: vrserver.txt should have Loaded server driver camvr. Re-run driver/install.ps1, restart SteamVR.

Project Structure

tracker/          # Python CV pipeline (app.py, bench.py, utils/pose.py, one_euro.py, ipc.py)
driver/           # OpenVR C++ driver (src/driver.cpp, tracked_device.cpp, CMakeLists.txt)
driver/dist/      # prebuilt driver (committed)
config/config.yaml
tools/            # fake_driver, check_udp, test_camera, download_models, test_vr_trackers
.venv/            # venv on D: (not committed)
run.bat / build_driver.bat

Roadmap

• Feet (heel) vs ankle offset tuning
• Tray UI