FCE-3838: React Native camera effects on Fishjam's own camera

.gitignore

@@ -140,5 +140,7 @@ docs/api/server/**
 docs/api/server-python/**
 docs/api/custom-video-source/**
 docs/api/vision-camera-source/**
+docs/api/video-effects/**
+docs/api/react-native-worklets/**
 
 .cursor/

.gitmodules

@@ -19,3 +19,9 @@
 [submodule "api/composition"]
 	path = api/composition
 	url = git@github.com:fishjam-cloud/foundry.git
+[submodule "packages/video-effects"]
+	path = packages/video-effects
+	url = https://github.com/fishjam-cloud/video-effects.git
+[submodule "packages/react-native-worklets"]
+	path = packages/react-native-worklets
+	url = https://github.com/fishjam-cloud/fishjam-react-native-worklets.git

docs/examples/react-native.mdx

@@ -115,6 +115,10 @@ The blur hook from `useBackgroundBlur` provides a `middleware` function that is
 
 Browse the full source: [blur-example on GitHub](https://github.com/fishjam-cloud/examples/tree/main/mobile-react-native/blur-example)
 
+:::tip
+For new apps, use `@fishjam-cloud/video-effects`, which blurs or replaces the background on the GPU on both iOS and Android. See [Background effects](../how-to/client/camera-effects/background-effects.mdx) and the [background blur tutorial](../tutorials/background-blur.mdx).
+:::
+
 ---
 
 ## Text Chat

docs/explanation/camera-effects.mdx

@@ -0,0 +1,100 @@
+---
+title: "How camera effects work"
+sidebar_position: 9.5
+description: Concepts behind Fishjam camera effects in React Native, including track middleware, the camera frame tap, the worklet runtime and the WebGPU render pipeline.
+---
+
+# How camera effects work
+
+[Camera effects](../how-to/client/camera-effects/index.mdx) change the camera video a React Native app publishes: a blurred background, a background image, or anything your own shaders draw. This page explains where effects plug into the Fishjam client, how camera frames reach a worklet and the GPU without copying pixels, and how effects such as background blur are structured.
+
+## Middleware, custom source, or VisionCamera?
+
+Fishjam clients offer three ways to publish processed camera video:
+
+| Approach                                                                           | What it does                                                                     | When to use it                                                                  |
+| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
+| [Camera track middleware](../how-to/client/camera-effects/stream-middleware.mdx)   | Replaces the camera track Fishjam captures with a processed one before it's sent | Effects on the regular camera; peers receive them as your `cameraTrack`         |
+| [Custom sources](../how-to/client/custom-sources/index.mdx)                        | Publishes a `MediaStream` your app produced on its own as a `customVideo` track  | Content that isn't the managed camera: renderers, compositors, file playback    |
+| [VisionCamera integration](../integrations/vision-camera/vision-camera-source.mdx) | Publishes a VisionCamera feed as a custom source                                 | Your app already depends on VisionCamera for capture, device control or plugins |
+
+Middleware is the default for effects: device selection, permissions, starting, stopping and switching cameras stay with [`useCamera`](../api/mobile/functions/useCamera), and the receiving side needs no changes.
+
+## Where effects plug in
+
+A camera track middleware is a function from the raw camera track to the track that is published instead. The Fishjam client owns when it runs:
+
+- **Every new device track goes through it.** The middleware runs when the camera starts, restarts or is switched, so the raw track is never published while a middleware is set.
+- **Publish first, release second.** When a middleware is set or replaced, the client builds the new track, swaps it into the published stream, and only then releases the previous one. Releasing a React Native track disposes it natively, so the order keeps the published stream from ever holding a dead track.
+- **The latest request wins.** If a middleware is replaced while it is still setting up, it releases itself as soon as it finishes and is never published. Until a new track is ready, peers keep receiving the previous one.
+
+The middleware and its `onClear` live in the client's camera state, not in a component, which is why an effect stays on across screens until it is cleared with `null`.
+
+## The camera frame pipeline
+
+An effect needs every camera frame, on the GPU, without slowing down the JS thread. React Native has no built-in way to do that, so the pipeline is split across three packages:
+
+```mermaid
+flowchart LR
+  camera["Camera capturer<br/>(react-native-webrtc)"] --> tap["Camera frame processor<br/>admits one frame at a time"]
+  tap --> runtime["Camera frame runtime<br/>(react-native-worklets)"]
+  runtime --> kernel["Your frame kernel<br/>or a video effect"]
+  kernel --> gpu["WebGPU render into<br/>a pooled output surface"]
+  gpu --> track["Pooled video track"]
+  track --> room["Published as<br/>your cameraTrack"]
+```
+
+### The camera tap
+
+`@fishjam-cloud/react-native-webrtc` exposes a **camera frame processor** for every local camera track, returned by `getCameraFrameProcessor`. A consumer attached to it receives the frames the camera captures, as native buffers: `CVPixelBufferRef`s on iOS and `AHardwareBuffer`s on Android.
+
+Frames are admitted one at a time. While a consumer still holds a frame, newer frames are dropped on the capture thread instead of being queued, so a slow effect lowers its own frame rate but never builds up latency.
+
+### The worklet runtime
+
+`@fishjam-cloud/react-native-worklets` is the consumer. `attachCameraFrameCallback` calls a worklet for every admitted frame on a dedicated **camera frame runtime**: a separate JS runtime with its own thread, created once for the whole app. The JS thread is not involved per frame, and the buffer handoff is a synchronous JSI call, which is why the pipeline requires the New Architecture.
+
+A worklet's closure is copied to that runtime when it is attached. Anything the per-frame code needs, such as GPU pipelines, bind groups and plain numbers, must be created up front on the JS thread and captured.
+
+### The render pipeline
+
+`createCameraFrameProcessorSession` in `@fishjam-cloud/video-effects` turns the tap into a published track:
+
+- It allocates a pool of three GPU-shareable output surfaces and a custom video track that reads from them. One surface can be encoding while the next is being drawn.
+- For every frame, it imports the camera buffer into WebGPU as an external texture, already rotated upright, and hands your frame kernel a context with that texture, a command encoder and the next output surface.
+- After the kernel has encoded its passes, it submits them once and pushes the surface to the track with a GPU fence, so the video encoder waits for the drawing to finish instead of reading a half-drawn frame. The frame keeps the camera's timestamp.
+
+Pixels stay in native GPU memory from the camera to the encoder; only handles move between the camera, the worklet and the track. The session's track is what the middleware returns, and its `dispose` detaches the tap and frees the pool.
+
+## Effects, sessions and frame kernels
+
+The ready-made effects use the same pipeline. An effect is described in three layers:
+
+- A **video effect** (`createBackgroundBlurEffect`, `createBackgroundImageEffect`) is a plain descriptor: an ID, the segmentation provider it needs, and a function that reads its options.
+- Creating the effect for a device builds a **session** on the JS thread: GPU textures, pipelines and, for background effects, the prepared segmentation model.
+- The session exposes a **frame kernel**: its GPU objects as plain data, plus worklet functions that encode one frame. Plain data is what lets the kernel be copied into the camera frame runtime.
+
+`createCameraEffectMiddleware` connects the layers: when the middleware runs it creates the session, captures its frame kernel and the effect's current options, and starts a frame processor session whose kernel calls the effect for every frame. Because the kernel works on a copy, options are read once, when the middleware is applied.
+
+### Segmentation
+
+Background effects need to know which pixels belong to the person. The segmentation provider from `typeGpuPersonSegmentation` runs a selfie segmentation model entirely on the GPU, written with [TypeGPU](https://docs.swmansion.com/TypeGPU/):
+
+- The model file is loaded and parsed once per model source and shared by every session that uses the same provider; each session builds its own GPU pipelines for it.
+- For each frame, the effect first offers the camera texture to the provider, which encodes the inference into the same command encoder as the effect. The mask the effect composites with therefore belongs to the same frame.
+- The effect then blurs the background, or draws the image behind the person, and softens the outline with the mask. When no recent mask is available, for example right after the model has loaded, the frame is published unchanged.
+
+If the model can't be loaded, the session is still created and reports an `"error"` status, and the effect publishes the camera as it is. A failure to create the session at all, for example on a device without the required GPU features, rejects the middleware instead.
+
+## Platform foundations
+
+- **iOS**: camera buffers are imported directly as multi-planar external textures, and output surfaces are IOSurface-backed BGRA8 (`bgra8unorm`). The shared GPU device is requested with the `dawn-multi-planar-formats` feature.
+- **Android**: the camera tap converts each camera image to an RGBA `AHardwareBuffer` before handing it over, and output surfaces are RGBA8 (`rgba8unorm`). Hardware buffers require Android 8.0 (API 26).
+- On both platforms, frame kernels sample the camera as RGB, so shaders written against `createCameraShaderBindings` with `cameraPixelLayout: "rgb"` work unchanged.
+
+## Where to go next
+
+- [Camera effects how-to guides](../how-to/client/camera-effects/index.mdx): background effects, your own WebGPU effects, frame processing and middleware
+- [Background blur tutorial](../tutorials/background-blur.mdx) and [WebGPU effects tutorial](../tutorials/webgpu-effects.mdx)
+- [How custom sources work](./custom-sources.mdx): the custom video track pipeline the render sessions build on
+- API reference: [Video Effects package](../api/video-effects/index.md), [React Native Worklets package](../api/react-native-worklets/index.md)

docs/explanation/custom-sources.mdx

@@ -12,11 +12,11 @@ description: Concepts behind Fishjam custom sources, including track metadata ro
 
 Fishjam clients offer three ways to influence what a peer publishes:
 
-| Approach                                                          | What it does                                                         | When to use it                                                            |
-| ----------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- |
-| Built-in sources (`useCamera`, `useMicrophone`, `useScreenShare`) | Capture and publish a device the SDK manages end-to-end              | Plain camera, microphone or screen sharing                                |
-| [Track middleware](../how-to/client/stream-middleware)            | Transforms a built-in track before it is sent (e.g. background blur) | You want the SDK's device handling, but with a processing step in between |
-| [Custom sources](../how-to/client/custom-sources/index.mdx)       | Publishes a `MediaStream` your app produced entirely on its own      | Content that isn't a managed device: renderers, compositors, ML pipelines |
+| Approach                                                              | What it does                                                         | When to use it                                                            |
+| --------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| Built-in sources (`useCamera`, `useMicrophone`, `useScreenShare`)     | Capture and publish a device the SDK manages end-to-end              | Plain camera, microphone or screen sharing                                |
+| [Track middleware](../how-to/client/camera-effects/stream-middleware) | Transforms a built-in track before it is sent (e.g. background blur) | You want the SDK's device handling, but with a processing step in between |
+| [Custom sources](../how-to/client/custom-sources/index.mdx)           | Publishes a `MediaStream` your app produced entirely on its own      | Content that isn't a managed device: renderers, compositors, ML pipelines |
 
 Capabilities differ slightly per platform:
 
@@ -27,6 +27,8 @@ Capabilities differ slightly per platform:
 | Custom audio                           | ✅ any audio track                 | ✅ PCM pushed through `useCustomAudioSource`        |
 | GPU rendering into the published video | ✅ WebGPU/WebGL via canvas capture | ✅ WebGPU toolkit with zero-copy camera import      |
 
+In React Native, effects on the camera, such as background blur or your own WebGPU shaders, run as track middleware on Fishjam's own camera; see [How camera effects work](./camera-effects.mdx). The custom video track pipeline described below is what those effects publish through.
+
 ## Streams and source IDs
 
 A custom source is a `MediaStream` registered under a stable **source ID** with `useCustomSource(sourceId)`. The ID makes the source addressable:
@@ -86,5 +88,7 @@ The native layer re-paces pushes into the continuous real-time frame stream the
 
 ## Where to go next
 
-- [Custom sources how-to guides](../how-to/client/custom-sources/index.mdx): publish from the web, React Native, Vision Camera, WebGPU, or your own pipeline
+- [Custom sources how-to guides](../how-to/client/custom-sources/index.mdx): publish from the web, React Native, or your own pipeline
+- [How camera effects work](./camera-effects.mdx): effects on Fishjam's own camera
+- [VisionCamera integration](../integrations/vision-camera/vision-camera-source.mdx): publish a VisionCamera feed
 - API reference: [`useCustomSource` (Web)](../api/web/functions/useCustomSource), [`useCustomSource` (React Native)](../api/mobile/functions/useCustomSource), [`useCustomAudioSource` (React Native)](../api/mobile/functions/useCustomAudioSource), [Vision Camera Source package](../api/vision-camera-source/index), [Custom Video Source package](../api/custom-video-source/index)

docs/how-to/client/camera-effects/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Camera effects",
+  "position": 7,
+  "link": {
+    "type": "doc",
+    "id": "how-to/client/camera-effects/index"
+  }
+}