Skip to content

background blur changes + examples - #606

Open
MiloszFilimowski wants to merge 10 commits into
mainfrom
fix/react-client-middleware-reapply
Open

background blur changes + examples#606
MiloszFilimowski wants to merge 10 commits into
mainfrom
fix/react-client-middleware-reapply

Conversation

@MiloszFilimowski

@MiloszFilimowski MiloszFilimowski commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Background blur in the fishjam-chat example, plus the react-client fixes it needs:

  1. Re-apply the camera middleware to new device tracks (react-client). A middleware set before the camera started was never applied, and a camera switch published the raw track for a moment. Now every new device track goes through the middleware before it is published, and a middleware that finishes setting up after it was replaced releases itself.
  2. Release the previous middleware only after the published track is swapped (react-client). Switching blur off used to release the blurred track first and swap it out of the published stream second. On React Native, releasing disposes the track natively, so the swap could no longer remove it: the dead track stayed in the stream and the local tile froze on every toggle (Android). applyMiddleware now builds the new track, publishes it through a callback from the caller, and releases the old one last. The latest request wins: one that is replaced while it sets up releases itself.
  3. Background blur in fishjam-chat (example). A blur button on the preview and room screens hands a middleware from @fishjam-cloud/video-effects to useCamera().setCameraTrackMiddleware. No provider, no custom source:
export const backgroundBlur = createCameraEffectMiddleware(
  createBackgroundBlurEffect(() => ({ segmentation, radius: 24 })),
);

const { currentCameraMiddleware, setCameraTrackMiddleware } = useCamera();
const isBlurOn = currentCameraMiddleware === backgroundBlur;
setCameraTrackMiddleware(isBlurOn ? null : backgroundBlur);

The model is read through expo-asset and expo-file-system, because an Android release build cannot fetch a bundled asset. The example moves to Expo SDK 57 / React Native 0.86.3, which react-native-worklets 0.12 requires, and sets iphoneDeploymentTarget: "16.4" in its Fishjam plugin options because Expo SDK 57 pods need it. The plugin default stays 15.1.

Motivation and Context

Camera effects on Fishjam's own camera track, shown in the example with the smallest possible app code. The react-client fixes came out of testing it on devices.

Uses @fishjam-cloud/video-effects 0.1.3 (fishjam-cloud/video-effects#4), which adds createCameraEffectMiddleware.

Related fork fix, released in @fishjam-cloud/react-native-webrtc 0.30.4 and not required by this PR: fishjam-cloud/fishjam-react-native-webrtc#91 (a released track leaves every local stream).

Checked

  • react-client unit tests: 77 pass; cameraMiddleware.spec.ts covers cold start, camera off and on, replacement and clearing while pending, onClear running once, and the previous middleware being released after replaceTrack.
  • Galaxy S23, debug and release builds: blur on and off on the preview and in the room, published tile blurred, room tile stays live across toggles.
  • iPhone: same on the preview and in the room.
  • Two phones in one room: each side's blur shows on the other phone.

Documentation impact

  • Documentation update required
  • Documentation updated in another PR
  • No documentation update required

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to
    not work as expected)

https://claude.ai/code/session_01MUzB8Nef5pJqJMBoPHgJNJ

… the raw one

A middleware was applied once, at the moment `setTrackMiddleware` was called, and
never again. The device track is replaced far more often than that — camera off
and on, a camera switch, a track that ended — and on each of those the effect
silently disappeared. Setting a middleware before starting the camera, which is
the natural order for a hook, meant it was stored and never applied at all.

Three fixes, all in the shared react-client path so web and React Native get them:

- Re-apply `currentMiddleware` when the raw device track changes. An `appliedTo`
  ref, claimed before the middleware is awaited, keeps the effect from racing a
  caller that is already applying it to the same track.
- Process a freshly acquired track before publishing it. `selectDevice` and
  `toggleDevice` handed the raw track straight to `replaceTrack`/`addTrack`, so
  unprocessed camera video went on the wire for as long as the middleware took to
  start — seconds, for an effect that loads a model. `applyMiddlewareToTrack`
  takes the track explicitly, because `rawTrack` still belongs to the previous
  device until React re-renders.
- Make release idempotent. `cleanupRef` was called but never cleared, so a stopped
  device followed by `setTrackMiddleware(null)` ran a consumer's `onClear` twice,
  freeing resources it no longer owned.

Tests cover cold start, off/on, double-clear and the publish path; four of the
five fail against the previous implementation.

A middleware that finishes setting up after it was replaced or cleared now releases itself:

A middleware whose setup was still pending when the next apply or a clear arrived had no cleanup
registered yet, so it was never released; once it resolved it overwrote the newer middleware's
track and cleanup. Each apply now carries a generation: a superseded result releases its own
track and yields whatever replaced it.
…d track is swapped

On React Native a middleware's onClear may dispose its track natively. Running
it before replaceTrack left the published stream holding a dead track it could
no longer remove, so the swap failed and the local tile froze.

applyMiddleware / applyMiddlewareToTrack now detach the previous middleware
and hand its release back to the caller as `releasePrevious`; the track
manager calls it only after replaceTrack (or right away when nothing is
published). The superseded-generation and stopped-device paths still release
immediately.

Claude-Session: https://claude.ai/code/session_01MUzB8Nef5pJqJMBoPHgJNJ
Expo SDK 57 pods require iOS 16.4, so `expo prebuild` with the 15.1 default
failed with "Specs satisfying the Expo dependency were found, but they
required a higher minimum deployment target". `ios.iphoneDeploymentTarget`
still overrides it.

Claude-Session: https://claude.ai/code/session_01MUzB8Nef5pJqJMBoPHgJNJ
A blur button on the preview and room screens hands a video-effects
middleware to useCamera().setCameraTrackMiddleware, so the effect runs on
the track the app already publishes and stays on across screens.

Expo moves to SDK 57 / React Native 0.86.3 because react-native-worklets
0.12 needs React Native 0.83 or newer.
@MiloszFilimowski MiloszFilimowski changed the title fix(react-client): re-apply track middleware to new device tracks and never publish the raw one feat(fishjam-chat): background blur, with camera middleware fixes Sep 11, 2026
…the plugin

The plugin default applies to every app that uses the SDK, so it stays
at 15.1. The example needs 16.4 for Expo SDK 57 pods and asks for it
through the plugin's iphoneDeploymentTarget option.
applyMiddleware read the device track from React state and
applyMiddlewareToTrack took it as an argument. One function that always
takes the track covers both: callers with a freshly acquired track pass
it, setTrackMiddleware passes the device's raw track, now exposed as
rawDeviceTrack.
applyMiddleware now builds the new track, publishes it through a
callback the caller passes, and only then releases the previous one.
The latest request wins: one replaced while it sets up releases itself.
That replaces the generation counter, the applied-track and latest-apply
refs, and the releasePrevious hand-off through the track manager.
Named helpers to start and stop a middleware, descriptive names for the
running middleware and the latest request, explicit early returns, and
separate effects for a stopped device and a new device track.
@MiloszFilimowski MiloszFilimowski changed the title feat(fishjam-chat): background blur, with camera middleware fixes background blur changes + examples Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant