feat(react-native): createCameraEffectMiddleware and release-safe model loading - #4
Merged
Merged
Conversation
A camera-track middleware for useCamera().setCameraTrackMiddleware. It acquires the GPU device, builds the effect session and the frame kernel, and disposes them when cleared, so an app can switch an effect on from any screen with no provider and no hook.
The plan is fetched and parsed once per model URL and reused by every later session, so switching an effect on again or restarting the camera skips the download and the parse. A failed load is not cached.
…d by URL An asset embedded in an Android release build is exposed as a file:///android_res URL that fetch cannot read. The option supplies the bytes instead, and the parsed model is cached per loader like it is per URL.
MiloszFilimowski
force-pushed
the
feat/camera-effect-middleware
branch
from
September 11, 2026 11:57
a489aed to
836bfff
Compare
3 of 6 tasks
MiloszFilimowski
changed the title
Merged
MiloszFilimowski
added a commit
to fishjam-cloud/web-client-sdk
that referenced
this pull request
Sep 14, 2026
## Description
Background blur in the fishjam-chat example, plus the react-client fixes
it needs:
1. **Re-apply the camera middleware to new device tracks**
(react-client). A middleware set before the camera started was never
applied, and a camera switch published the raw track for a moment. Now
every new device track goes through the middleware before it is
published, and a middleware that finishes setting up after it was
replaced releases itself.
2. **Release the previous middleware only after the published track is
swapped** (react-client). Switching blur off used to release the blurred
track first and swap it out of the published stream second. On React
Native, releasing disposes the track natively, so the swap could no
longer remove it: the dead track stayed in the stream and the local tile
froze on every toggle (Android). `applyMiddleware` now builds the new
track, publishes it through a callback from the caller, and releases the
old one last. The latest request wins: one that is replaced while it
sets up releases itself.
3. **Background blur in fishjam-chat** (example). A blur button on the
preview and room screens hands a middleware from
`@fishjam-cloud/video-effects` to
`useCamera().setCameraTrackMiddleware`. No provider, no custom source:
```ts
export const backgroundBlur = createCameraEffectMiddleware(
createBackgroundBlurEffect(() => ({ segmentation, radius: 24 })),
);
const { currentCameraMiddleware, setCameraTrackMiddleware } = useCamera();
const isBlurOn = currentCameraMiddleware === backgroundBlur;
setCameraTrackMiddleware(isBlurOn ? null : backgroundBlur);
```
The model is read through `expo-asset` and `expo-file-system`, because
an Android release build cannot `fetch` a bundled asset. The example
moves to Expo SDK 57 / React Native 0.86.3, which react-native-worklets
0.12 requires, and sets `iphoneDeploymentTarget: "16.4"` in its Fishjam
plugin options because Expo SDK 57 pods need it. The plugin default
stays 15.1.
## Motivation and Context
Camera effects on Fishjam's own camera track, shown in the example with
the smallest possible app code. The react-client fixes came out of
testing it on devices.
Uses `@fishjam-cloud/video-effects` 0.1.3
(fishjam-cloud/video-effects#4), which adds
`createCameraEffectMiddleware`.
Related fork fix, released in `@fishjam-cloud/react-native-webrtc`
0.30.4 and not required by this PR:
fishjam-cloud/fishjam-react-native-webrtc#91 (a released track leaves
every local stream).
## Checked
- react-client unit tests: 77 pass; `cameraMiddleware.spec.ts` covers
cold start, camera off and on, replacement and clearing while pending,
onClear running once, and the previous middleware being released after
`replaceTrack`.
- Galaxy S23, debug and release builds: blur on and off on the preview
and in the room, published tile blurred, room tile stays live across
toggles.
- iPhone: same on the preview and in the room.
- Two phones in one room: each side's blur shows on the other phone.
## Documentation impact
- [ ] Documentation update required
- [ ] Documentation updated [in another PR](_)
- [x] No documentation update required
## Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to
not work as expected)
https://claude.ai/code/session_01MUzB8Nef5pJqJMBoPHgJNJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Camera effect as a plain middleware, plus model loading that works in release builds
Three commits:
createCameraEffectMiddleware(effect, options?)in./fishjam-react-native. It returns a FishjamTrackMiddleware, so an app switches an effect on with one call, no provider or hook:It builds the effect session and the camera frame processor on the raw track and tears both down in
onClear.onStatusreports loading and failures. Also exportsgetCameraWebGpuDevice(), the shared device it uses.The parsed segmentation model is shared between sessions. It is fetched and parsed once per model URL, so switching the effect on again or restarting the camera skips the download and the parse. A failed load is not cached.
typeGpuPersonSegmentation({ loadModel })supplies the model bytes instead of fetchingmodelUrl. An Android release build exposes a bundled asset as afile:///android_res/...URL that React Native'sfetchcannot read, so the effect silently fell back to passthrough there. The parsed model is cached per loader as well.README and CHANGELOG are updated.
useFishjamCameraEffectis unchanged.Checked
yarn test(tsc, eslint, prettier) andyarn build. On devices through the fishjam-chat example (fishjam-cloud/web-client-sdk#606): Galaxy S23 debug and release builds, iPhone, blur on and off on the preview and in the room, and two phones in one room seeing each other blurred.Releasing this as 0.1.3 unblocks the example PR.
https://claude.ai/code/session_01MUzB8Nef5pJqJMBoPHgJNJ