# rvm-matting

A standalone Web Worker that runs [Robust Video Matting](https://github.com/PeterL1n/RobustVideoMatting)
(RVM) in the browser with [ONNX Runtime Web](https://onnxruntime.ai/). Send it
RGBA video frames; it sends back alpha mattes. It is a self-contained program
with a generic message protocol and no knowledge of the app that uses it.

Licensed under the **GNU General Public License v3.0** (see `LICENSE`), as is RVM.

## Attribution

Robust Video Matting by Shanchuan Lin, Linjie Yang, Imran Saleemi and
Soumyadip Sengupta — *Robust High-Resolution Video Matting with Temporal
Guidance*, WACV 2022. Weights: the official `rvm_mobilenetv3_fp32.onnx`
release (v1.0.0), GPL-3.0.

ONNX Runtime Web is © Microsoft Corporation, MIT licence; its runtime files
are redistributed unmodified.

## Modification

`scripts/prepare.mjs` changes three bytes of the official ONNX file: the
model's three 2×2, stride-2 `AveragePool` nodes are switched from
`ceil_mode=1` to `ceil_mode=0`, because ONNX Runtime's WebGPU backend does not
support ceil mode for pooling. The output is identical whenever the pooled
sizes are even. Both the original and patched checksums are verified.

## Building

```sh
node scripts/prepare.mjs --out <dir> --ort <path to onnxruntime-web/dist>
node scripts/check.mjs --out <dir>
```

Serve `<dir>` as static files and start `<dir>/worker.js` as a module worker.

## Protocol

```js
worker.postMessage({ type: "init" });
// ← { type: "ready", backend: "webgpu" | "wasm" } or { type: "error", message }

worker.postMessage(
	{ type: "matte", id, rgba, width, height, stream, downsampleRatio },
	[rgba.buffer],
);
// ← { type: "matte", id, alpha: Float32Array (0–1), width, height }
//   or { type: "error", id, message }

worker.postMessage({ type: "end-stream", stream });

// Optional: let another worker talk to this one directly.
const { port1, port2 } = new MessageChannel();
worker.postMessage({ type: "connect", port: port1 }, [port1]);
otherWorker.postMessage({ rvmPort: port2 }, [port2]); // then use port2
```

Frames that share a `stream` id share RVM's recurrent memory, which keeps the
matte temporally stable, so send a clip's frames in order. A frame without a
stream is run a few times on itself to settle the memory. `downsampleRatio`
defaults to a value that puts RVM's coarse pass at about 256 px on the short
edge.