diff --git a/.gitignore b/.gitignore index c6065300..9eb2b0f1 100644 --- a/.gitignore +++ b/.gitignore @@ -140,5 +140,7 @@ docs/api/server/** docs/api/server-python/** docs/api/custom-video-source/** docs/api/vision-camera-source/** +docs/api/video-effects/** +docs/api/react-native-worklets/** .cursor/ diff --git a/.gitmodules b/.gitmodules index f9ec1a7a..d1480a8a 100644 --- a/.gitmodules +++ b/.gitmodules @@ -19,3 +19,9 @@ [submodule "api/composition"] path = api/composition url = git@github.com:fishjam-cloud/foundry.git +[submodule "packages/video-effects"] + path = packages/video-effects + url = https://github.com/fishjam-cloud/video-effects.git +[submodule "packages/react-native-worklets"] + path = packages/react-native-worklets + url = https://github.com/fishjam-cloud/fishjam-react-native-worklets.git diff --git a/docs/examples/react-native.mdx b/docs/examples/react-native.mdx index 762dc616..9bda5114 100644 --- a/docs/examples/react-native.mdx +++ b/docs/examples/react-native.mdx @@ -115,6 +115,10 @@ The blur hook from `useBackgroundBlur` provides a `middleware` function that is Browse the full source: [blur-example on GitHub](https://github.com/fishjam-cloud/examples/tree/main/mobile-react-native/blur-example) +:::tip +For new apps, use `@fishjam-cloud/video-effects`, which blurs or replaces the background on the GPU on both iOS and Android. See [Background effects](../how-to/client/camera-effects/background-effects.mdx) and the [background blur tutorial](../tutorials/background-blur.mdx). +::: + --- ## Text Chat diff --git a/docs/explanation/camera-effects.mdx b/docs/explanation/camera-effects.mdx new file mode 100644 index 00000000..71ac8c3a --- /dev/null +++ b/docs/explanation/camera-effects.mdx @@ -0,0 +1,100 @@ +--- +title: "How camera effects work" +sidebar_position: 9.5 +description: Concepts behind Fishjam camera effects in React Native, including track middleware, the camera frame tap, the worklet runtime and the WebGPU render pipeline. +--- + +# How camera effects work + +[Camera effects](../how-to/client/camera-effects/index.mdx) change the camera video a React Native app publishes: a blurred background, a background image, or anything your own shaders draw. This page explains where effects plug into the Fishjam client, how camera frames reach a worklet and the GPU without copying pixels, and how effects such as background blur are structured. + +## Middleware, custom source, or VisionCamera? + +Fishjam clients offer three ways to publish processed camera video: + +| Approach | What it does | When to use it | +| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| [Camera track middleware](../how-to/client/camera-effects/stream-middleware.mdx) | Replaces the camera track Fishjam captures with a processed one before it's sent | Effects on the regular camera; peers receive them as your `cameraTrack` | +| [Custom sources](../how-to/client/custom-sources/index.mdx) | Publishes a `MediaStream` your app produced on its own as a `customVideo` track | Content that isn't the managed camera: renderers, compositors, file playback | +| [VisionCamera integration](../integrations/vision-camera/vision-camera-source.mdx) | Publishes a VisionCamera feed as a custom source | Your app already depends on VisionCamera for capture, device control or plugins | + +Middleware is the default for effects: device selection, permissions, starting, stopping and switching cameras stay with [`useCamera`](../api/mobile/functions/useCamera), and the receiving side needs no changes. + +## Where effects plug in + +A camera track middleware is a function from the raw camera track to the track that is published instead. The Fishjam client owns when it runs: + +- **Every new device track goes through it.** The middleware runs when the camera starts, restarts or is switched, so the raw track is never published while a middleware is set. +- **Publish first, release second.** When a middleware is set or replaced, the client builds the new track, swaps it into the published stream, and only then releases the previous one. Releasing a React Native track disposes it natively, so the order keeps the published stream from ever holding a dead track. +- **The latest request wins.** If a middleware is replaced while it is still setting up, it releases itself as soon as it finishes and is never published. Until a new track is ready, peers keep receiving the previous one. + +The middleware and its `onClear` live in the client's camera state, not in a component, which is why an effect stays on across screens until it is cleared with `null`. + +## The camera frame pipeline + +An effect needs every camera frame, on the GPU, without slowing down the JS thread. React Native has no built-in way to do that, so the pipeline is split across three packages: + +```mermaid +flowchart LR + camera["Camera capturer
(react-native-webrtc)"] --> tap["Camera frame processor
admits one frame at a time"] + tap --> runtime["Camera frame runtime
(react-native-worklets)"] + runtime --> kernel["Your frame kernel
or a video effect"] + kernel --> gpu["WebGPU render into
a pooled output surface"] + gpu --> track["Pooled video track"] + track --> room["Published as
your cameraTrack"] +``` + +### The camera tap + +`@fishjam-cloud/react-native-webrtc` exposes a **camera frame processor** for every local camera track, returned by `getCameraFrameProcessor`. A consumer attached to it receives the frames the camera captures, as native buffers: `CVPixelBufferRef`s on iOS and `AHardwareBuffer`s on Android. + +Frames are admitted one at a time. While a consumer still holds a frame, newer frames are dropped on the capture thread instead of being queued, so a slow effect lowers its own frame rate but never builds up latency. + +### The worklet runtime + +`@fishjam-cloud/react-native-worklets` is the consumer. `attachCameraFrameCallback` calls a worklet for every admitted frame on a dedicated **camera frame runtime**: a separate JS runtime with its own thread, created once for the whole app. The JS thread is not involved per frame, and the buffer handoff is a synchronous JSI call, which is why the pipeline requires the New Architecture. + +A worklet's closure is copied to that runtime when it is attached. Anything the per-frame code needs, such as GPU pipelines, bind groups and plain numbers, must be created up front on the JS thread and captured. + +### The render pipeline + +`createCameraFrameProcessorSession` in `@fishjam-cloud/video-effects` turns the tap into a published track: + +- It allocates a pool of three GPU-shareable output surfaces and a custom video track that reads from them. One surface can be encoding while the next is being drawn. +- For every frame, it imports the camera buffer into WebGPU as an external texture, already rotated upright, and hands your frame kernel a context with that texture, a command encoder and the next output surface. +- After the kernel has encoded its passes, it submits them once and pushes the surface to the track with a GPU fence, so the video encoder waits for the drawing to finish instead of reading a half-drawn frame. The frame keeps the camera's timestamp. + +Pixels stay in native GPU memory from the camera to the encoder; only handles move between the camera, the worklet and the track. The session's track is what the middleware returns, and its `dispose` detaches the tap and frees the pool. + +## Effects, sessions and frame kernels + +The ready-made effects use the same pipeline. An effect is described in three layers: + +- A **video effect** (`createBackgroundBlurEffect`, `createBackgroundImageEffect`) is a plain descriptor: an ID, the segmentation provider it needs, and a function that reads its options. +- Creating the effect for a device builds a **session** on the JS thread: GPU textures, pipelines and, for background effects, the prepared segmentation model. +- The session exposes a **frame kernel**: its GPU objects as plain data, plus worklet functions that encode one frame. Plain data is what lets the kernel be copied into the camera frame runtime. + +`createCameraEffectMiddleware` connects the layers: when the middleware runs it creates the session, captures its frame kernel and the effect's current options, and starts a frame processor session whose kernel calls the effect for every frame. Because the kernel works on a copy, options are read once, when the middleware is applied. + +### Segmentation + +Background effects need to know which pixels belong to the person. The segmentation provider from `typeGpuPersonSegmentation` runs a selfie segmentation model entirely on the GPU, written with [TypeGPU](https://docs.swmansion.com/TypeGPU/): + +- The model file is loaded and parsed once per model source and shared by every session that uses the same provider; each session builds its own GPU pipelines for it. +- For each frame, the effect first offers the camera texture to the provider, which encodes the inference into the same command encoder as the effect. The mask the effect composites with therefore belongs to the same frame. +- The effect then blurs the background, or draws the image behind the person, and softens the outline with the mask. When no recent mask is available, for example right after the model has loaded, the frame is published unchanged. + +If the model can't be loaded, the session is still created and reports an `"error"` status, and the effect publishes the camera as it is. A failure to create the session at all, for example on a device without the required GPU features, rejects the middleware instead. + +## Platform foundations + +- **iOS**: camera buffers are imported directly as multi-planar external textures, and output surfaces are IOSurface-backed BGRA8 (`bgra8unorm`). The shared GPU device is requested with the `dawn-multi-planar-formats` feature. +- **Android**: the camera tap converts each camera image to an RGBA `AHardwareBuffer` before handing it over, and output surfaces are RGBA8 (`rgba8unorm`). Hardware buffers require Android 8.0 (API 26). +- On both platforms, frame kernels sample the camera as RGB, so shaders written against `createCameraShaderBindings` with `cameraPixelLayout: "rgb"` work unchanged. + +## Where to go next + +- [Camera effects how-to guides](../how-to/client/camera-effects/index.mdx): background effects, your own WebGPU effects, frame processing and middleware +- [Background blur tutorial](../tutorials/background-blur.mdx) and [WebGPU effects tutorial](../tutorials/webgpu-effects.mdx) +- [How custom sources work](./custom-sources.mdx): the custom video track pipeline the render sessions build on +- API reference: [Video Effects package](../api/video-effects/index.md), [React Native Worklets package](../api/react-native-worklets/index.md) diff --git a/docs/explanation/custom-sources.mdx b/docs/explanation/custom-sources.mdx index ae979605..1cf6ed08 100644 --- a/docs/explanation/custom-sources.mdx +++ b/docs/explanation/custom-sources.mdx @@ -12,11 +12,11 @@ description: Concepts behind Fishjam custom sources, including track metadata ro Fishjam clients offer three ways to influence what a peer publishes: -| Approach | What it does | When to use it | -| ----------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- | -| Built-in sources (`useCamera`, `useMicrophone`, `useScreenShare`) | Capture and publish a device the SDK manages end-to-end | Plain camera, microphone or screen sharing | -| [Track middleware](../how-to/client/stream-middleware) | Transforms a built-in track before it is sent (e.g. background blur) | You want the SDK's device handling, but with a processing step in between | -| [Custom sources](../how-to/client/custom-sources/index.mdx) | Publishes a `MediaStream` your app produced entirely on its own | Content that isn't a managed device: renderers, compositors, ML pipelines | +| Approach | What it does | When to use it | +| --------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| Built-in sources (`useCamera`, `useMicrophone`, `useScreenShare`) | Capture and publish a device the SDK manages end-to-end | Plain camera, microphone or screen sharing | +| [Track middleware](../how-to/client/camera-effects/stream-middleware) | Transforms a built-in track before it is sent (e.g. background blur) | You want the SDK's device handling, but with a processing step in between | +| [Custom sources](../how-to/client/custom-sources/index.mdx) | Publishes a `MediaStream` your app produced entirely on its own | Content that isn't a managed device: renderers, compositors, ML pipelines | Capabilities differ slightly per platform: @@ -27,6 +27,8 @@ Capabilities differ slightly per platform: | Custom audio | βœ… any audio track | βœ… PCM pushed through `useCustomAudioSource` | | GPU rendering into the published video | βœ… WebGPU/WebGL via canvas capture | βœ… WebGPU toolkit with zero-copy camera import | +In React Native, effects on the camera, such as background blur or your own WebGPU shaders, run as track middleware on Fishjam's own camera; see [How camera effects work](./camera-effects.mdx). The custom video track pipeline described below is what those effects publish through. + ## Streams and source IDs A custom source is a `MediaStream` registered under a stable **source ID** with `useCustomSource(sourceId)`. The ID makes the source addressable: @@ -86,5 +88,7 @@ The native layer re-paces pushes into the continuous real-time frame stream the ## Where to go next -- [Custom sources how-to guides](../how-to/client/custom-sources/index.mdx): publish from the web, React Native, Vision Camera, WebGPU, or your own pipeline +- [Custom sources how-to guides](../how-to/client/custom-sources/index.mdx): publish from the web, React Native, or your own pipeline +- [How camera effects work](./camera-effects.mdx): effects on Fishjam's own camera +- [VisionCamera integration](../integrations/vision-camera/vision-camera-source.mdx): publish a VisionCamera feed - API reference: [`useCustomSource` (Web)](../api/web/functions/useCustomSource), [`useCustomSource` (React Native)](../api/mobile/functions/useCustomSource), [`useCustomAudioSource` (React Native)](../api/mobile/functions/useCustomAudioSource), [Vision Camera Source package](../api/vision-camera-source/index), [Custom Video Source package](../api/custom-video-source/index) diff --git a/docs/how-to/client/camera-effects/_category_.json b/docs/how-to/client/camera-effects/_category_.json new file mode 100644 index 00000000..6a3efaaf --- /dev/null +++ b/docs/how-to/client/camera-effects/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Camera effects", + "position": 7, + "link": { + "type": "doc", + "id": "how-to/client/camera-effects/index" + } +} diff --git a/docs/how-to/client/camera-effects/background-effects.mdx b/docs/how-to/client/camera-effects/background-effects.mdx new file mode 100644 index 00000000..5e4da79b --- /dev/null +++ b/docs/how-to/client/camera-effects/background-effects.mdx @@ -0,0 +1,367 @@ +--- +title: "Blur or replace the camera background" +sidebar_position: 1 +sidebar_label: "Background effects πŸ“±" +description: Blur or replace the background of the camera you publish to Fishjam, with ready-made effects from @fishjam-cloud/video-effects that run on the GPU. +--- + +# Blur or replace the camera background Mobile + +:::note +This guide is exclusively for **Mobile** (React Native) applications. +::: + +`@fishjam-cloud/video-effects` ships ready-made background effects: background blur and background image replacement. An effect runs on the camera track Fishjam already publishes, as a [camera track middleware](./stream-middleware.mdx). Your app keeps using [`useCamera`](../../../api/mobile/functions/useCamera), and other peers keep receiving your video as `peer.cameraTrack`, now with the effect applied. + +Each camera frame reaches a worklet on a dedicated camera thread, a segmentation model finds the person on the GPU, and the effect draws the result into the published frame. The JS thread is not involved per frame. See [How camera effects work](../../../explanation/camera-effects.mdx) for the details. + +:::tip[First time?] +The [background blur tutorial](../../../tutorials/background-blur.mdx) builds this step by step in a working app, from installing the packages to a blur toggle. +::: + +## Prerequisites + +- `@fishjam-cloud/react-native-client` **0.30.2** or newer, with your app wrapped in `FishjamProvider` (see [Installation](../installation.mdx)) +- `@fishjam-cloud/video-effects` **0.1.5** or newer +- `react-native-webgpu` **0.10.1** or newer. Older versions leak one camera frame of graphics memory per frame on Android until the JavaScript garbage collector runs. +- React Native 0.86 (Expo SDK 57) with the New Architecture enabled +- iOS 16.4 or newer, or Android 8.0 (API 26) or newer +- A physical device, or the iOS Simulator with a virtual camera from [SimCam](../ios-simulator-camera.mdx) + +## Install + +```bash npm2yarn +npm install @fishjam-cloud/video-effects @fishjam-cloud/react-native-worklets react-native-worklets react-native-webgpu expo-asset expo-file-system expo-build-properties +npm install --save-dev unplugin-typegpu @babel/plugin-transform-class-static-block +``` + +What each package does: + +- `@fishjam-cloud/video-effects`: the effects, the segmentation model and the camera middleware +- `@fishjam-cloud/react-native-worklets`: runs a worklet on every frame of the Fishjam camera track. Its minor version follows `react-native-worklets`: `0.12.x` works with `react-native-worklets` `0.12.x`. +- `react-native-webgpu`: the GPU the effects render with +- `expo-asset` and `expo-file-system`: load the bundled segmentation model +- `expo-build-properties`: raises the Android minimum SDK version + +### Configure Babel + +The effects rely on the `react-native-worklets` Babel plugin, and on TypeGPU, which needs `import.meta`, class static blocks and its own Babel plugin. Keep `react-native-worklets/plugin` as the last plugin: + +```js title='babel.config.js' +module.exports = function (api) { + api.cache(true); + return { + presets: [["babel-preset-expo", { unstable_transformImportMeta: true }]], + plugins: [ + "@babel/plugin-transform-class-static-block", + "unplugin-typegpu/babel", + "react-native-worklets/plugin", + ], + }; +}; +``` + +### Configure Metro + +The segmentation model ships as a `.ssgbin` file. Add the extension to Metro's asset extensions so it can be bundled with your app: + +```js title='metro.config.js' +const { getDefaultConfig } = require("expo/metro-config"); + +const config = getDefaultConfig(__dirname); + +config.resolver.assetExts = [...config.resolver.assetExts, "ssgbin"]; + +module.exports = config; +``` + +### Set the minimum OS versions + +Expo SDK 57 needs iOS 16.4, while the Fishjam config plugin sets 15.1 by default. `react-native-webgpu` needs Android 8.0 (API 26), while Expo defaults to API 24. Raise both in `app.json`, the Android one with `expo-build-properties`: + +```json title='app.json' +{ + "expo": { + "plugins": [ + [ + "@fishjam-cloud/react-native-client", + { + "ios": { + "iphoneDeploymentTarget": "16.4" + } + } + ], + [ + "expo-build-properties", + { + "android": { + "minSdkVersion": 26 + } + } + ] + ] + } +} +``` + +In a bare React Native app, set `platform :ios, '16.4'` in `ios/Podfile` and `minSdkVersion = 26` in `android/build.gradle` instead. + +Then restart Metro with a cleared cache and rebuild the native app, since the new packages contain native code: + +```bash +npx expo prebuild +npx expo run:ios # or run:android +``` + +## Load the segmentation model + +Both effects need a segmentation provider, which finds the person in each frame. `typeGpuPersonSegmentation` runs the bundled model on the GPU. Create the provider once, at module scope, so every effect that uses it shares one loaded model: + +```ts title='effects.ts' +import { typeGpuPersonSegmentation } from "@fishjam-cloud/video-effects/segmentation/typegpu"; +import { Asset } from "expo-asset"; +import { File } from "expo-file-system"; + +const segmentationModel = Asset.fromModule( + require("@fishjam-cloud/video-effects/assets/selfie_segmenter.ssgbin"), +); + +async function loadSegmentationModel(): Promise { + await segmentationModel.downloadAsync(); + if (!segmentationModel.localUri) { + throw new Error("The segmentation model is not available."); + } + return new File(segmentationModel.localUri).arrayBuffer(); +} + +export const segmentation = typeGpuPersonSegmentation({ + loadModel: loadSegmentationModel, +}); +``` + +The model is read from a local copy of the asset instead of being fetched by URL, because an Android release build cannot `fetch` a bundled asset. If you host the model yourself, pass its address as `modelUrl` instead of `loadModel`. + +## Blur the background + +Wrap the effect in `createCameraEffectMiddleware` and keep the middleware at module scope. A stable identity lets you tell whether it is active by comparing it with `currentCameraMiddleware`: + +```ts title='effects.ts' +import type { PersonSegmentationProvider } from "@fishjam-cloud/video-effects"; +declare const segmentation: PersonSegmentationProvider; +// ---cut--- +import { createBackgroundBlurEffect } from "@fishjam-cloud/video-effects/background-blur"; +import { createCameraEffectMiddleware } from "@fishjam-cloud/video-effects/fishjam-react-native"; + +export const backgroundBlur = createCameraEffectMiddleware( + createBackgroundBlurEffect(() => ({ segmentation, radius: 24 })), +); +``` + +Then switch it on and off with `setCameraTrackMiddleware`: + +```tsx +import type { TrackMiddleware } from "@fishjam-cloud/react-native-client"; +declare const backgroundBlur: NonNullable; +// ---cut--- +import React from "react"; +import { Button } from "react-native"; +import { useCamera } from "@fishjam-cloud/react-native-client"; + +export function BlurToggle() { + const { currentCameraMiddleware, setCameraTrackMiddleware } = useCamera(); + const isBlurOn = currentCameraMiddleware === backgroundBlur; + + const toggleBlur = async () => { + try { + await setCameraTrackMiddleware(isBlurOn ? null : backgroundBlur); + } catch (error) { + console.warn("Background blur failed", error); + await setCameraTrackMiddleware(null); + } + }; + + return ( + + + {cameraStream &&