Three.js Mixed Reality Compositor (Chroma Key & Camera Feed)

A mixed-reality video compositor in Three.js, with YCbCr chroma keying, spill suppression and a key matte view for green-screen actors in a virtual set, or virtual objects over a camera plate, colour-matched for exposure, white balance and grain, using a synthetic camera or your webcam.

Key topics: mixed reality compositing, chroma key shader, green screen, camera feed, webcam three.js, video compositing

What this code does

- Sources: the camera plate is a CanvasTexture, either an animated synthetic presenter on a lit green screen or a real webcam via getUserMedia.
- Keying: alpha comes from the distance between each pixel's CbCr chroma and the key colour's, with smoothstep softness, which is more stable than RGB distance under uneven lighting.
- Spill Suppression: green above max(red, blue) is pulled back, removing the green fringe on edges.
- Layers: the virtual scene renders into a render target with alpha, and one full-screen shader composites camera and CG.
- Colour Matching: exposure, white balance and grain applied to the CG layer make it sit in the plate.
- Controls: composite mode (keyed actor, overlay, key matte), threshold, softness, spill, exposure, white balance and grain.

Continue Learning

JavaScript Implementation

ES6+
// Mixed-reality compositor: camera plate + CG layer in one full-screen pass
const camTex = new THREE.VideoTexture(webcamVideo)            // or a CanvasTexture
renderer.setRenderTarget(cgTarget); renderer.setClearColor(0, 0); renderer.render(virtualScene, camera)
// fragment:
//   vec2 cbcr = chroma(cam);  float a = smoothstep(threshold, threshold + soft, distance(cbcr, chroma(key)));
//   cam.g -= max(0.0, cam.g - max(cam.r, cam.b)) * spill;     // remove green spill
//   gl_FragColor = vec4(mix(cg.rgb, cam, a), 1.0);              // keyed actor over the virtual set
//   // overlay mode instead: mix(cam, cg.rgb * exposure * whiteBalance, cg.a)