Eye and mouth expression tracking that runs on the Steam Frame itself.
Babble Frame Runtime reads the Steam Frame's eye cameras, and a wired Babble face tracker for the mouth, and runs a set of optimized neural networks to output expression values (eye openness, widening, pupil dilation; jaw, lips, cheeks and tongue), which it sends to VRCFaceTracking on your PC.
It is installed as a SteamVR driver for settings integration. The driver starts the runtime as a separate process and restarts it if it stops, so a problem in the runtime never affects SteamVR.
- Eye expressions: openness, widen and pupil dilation per eye.
- Mouth tracking: 43 blendshapes from a wired Babble face tracker plugged into the headset.
- Gaze: each eye's own gaze from the headset's eye service, or SteamVR's combined gaze.
- Presence: tracking pauses while the headset is off the head (proximity sensor), and the cameras turn off.
- SteamVR Settings: an Expression Tracking section to turn mouth and eye tracking on and off and set the eye tracking rate.
- Outputs:
- OSC in the Baballonia format for VRCFaceTracking.
- A plugin interface for other outputs.
- Dataset capture: an optional guided recording app, installed from the overlay's Diagnostics page.
Work in progress. The mouth and eye models are prototypes; the runtime keeps models swappable so better ones can be dropped in.
On the Steam Frame, switch to Desktop Mode, open Konsole and paste:
curl -fsSL https://raw.githubusercontent.com/Project-Babble/Babble-Frame-Runtime/main/tools/get.sh | bash
It downloads the newest release, checks it against the release's SHA-256 checksum, installs it in your home folder and restarts SteamVR. Its settings are in SteamVR Settings, under Expression Tracking; turn on Show App Window there for the Babble tab in the SteamVR dashboard, where updates appear next to the version.
To uninstall, run ~/.local/share/babble-frame-runtime/uninstall.sh.
- Get the source on a PC and copy it to the headset.
- Build ExecuTorch once, then build the driver, on the headset.
- Register the driver with
vrpathreg.sh, then restart SteamVR.
-DBABBLE_PROFILE=release (what releases ship) builds the USB Babble tracker as the only mouth and VRCFaceTracking as
the only output; it leaves out the Frame's front-port mouth cameras and their camera arbiter, the VRChat-native output,
the visualizer launcher, the camera-stream switch and the unused models. The default, dev, builds everything including unfinished features.
Licensed under the GNU General Public License v3.0. Third-party code in third_party/ keeps its own license.