JusperLee/Dolphin · Audio-Visual Speech Separation

Two voices, one track. Split by lips.

Dolphin separates a speaker from a mixture using the video: it reads lip movement to know who is talking. 7 M parameters, 6× faster than the previous state of the art. This page runs the released checkpoint entirely on your device — face tracking with MediaPipe, the model through WebGPU via ONNX Runtime Web. No server, no upload.

Paper Code Model card Official demo

Drop a file anywhere on the page, or pick one above. Everything happens locally — the video is never uploaded.

Pipeline

Separated speakers

Results appear here: one audio track per tracked face, with your choice of what the video plays along with.

How it works here

  1. Frames. The clip is walked at 25 fps (Dolphin's training rate); each frame goes to MediaPipe's face landmarker, up to four faces.
  2. Tracks. Detections are linked across frames into speaker tracks by box proximity, exactly the way inference.py matches boxes.
  3. Mouth ROI. Landmarks are remapped from the 478-point face mesh onto the 68-point layout, smoothed over a 12-frame window, aligned to the LRW mean face by similarity transform, then cropped to a 96×96 mouth patch and centre-cropped to 88×88 grey — the exact preprocessing the checkpoint was trained on.
  4. Separation. For each 4-second window and each tracked face, the mixed waveform and that face's mouth patches go through the ONNX graph. EXACT-DCT heat layers replace the dense 16000×16000 cosine matrices the paper's code builds, so the graph stays small.
  5. Assembly. Windows are overlap-added with a 1 s crossfade and gain-matched to the mixture, then written as 16 kHz mono WAV.

The weights are not re-hosted: the browser streams model.safetensors straight from the original repository and binds them to the graph as external data.

Licence and provenance

Source code in the Dolphin repository is Apache-2.0, but the released weights are not covered by that licence. The checkpoint's video encoder was distilled from AV-HuBERT feature targets, and the authors state they do not represent it as cleared for commercial or production use. Read MODEL_PROVENANCE.md and the AV-HuBERT licence before using the model. This demo is for research and evaluation, and runs the model at your own risk on your own machine.

Model: Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention, Kai Li, Kejun Gao, Xiaolin Hu — Audio and Speech Group, Tsinghua University. Browser port: ONNX Runtime Web (WebGPU) + MediaPipe Tasks.