Three.js Mixed Reality Compositor (Chroma Key & Camera Feed)
A mixed-reality video compositor in Three.js, with YCbCr chroma keying, spill suppression and a key matte view for green-screen actors in a virtual set, or virtual objects over a camera plate, colour-matched for exposure, white balance and grain, using a synthetic camera or your webcam.
Key topics: mixed reality compositing, chroma key shader, green screen, camera feed, webcam three.js, video compositing
Game Over
What this code does
- Sources: the camera plate is a CanvasTexture, either an animated synthetic presenter on a lit green screen or a real webcam via getUserMedia.
- Keying: alpha comes from the distance between each pixel's CbCr chroma and the key colour's, with smoothstep softness, which is more stable than RGB distance under uneven lighting.
- Spill Suppression: green above max(red, blue) is pulled back, removing the green fringe on edges.
- Layers: the virtual scene renders into a render target with alpha, and one full-screen shader composites camera and CG.
- Colour Matching: exposure, white balance and grain applied to the CG layer make it sit in the plate.
- Controls: composite mode (keyed actor, overlay, key matte), threshold, softness, spill, exposure, white balance and grain.
Related Demos
Continue Learning
JavaScript Implementation
// Mixed-reality compositor: camera plate + CG layer in one full-screen pass
const camTex = new THREE.VideoTexture(webcamVideo) // or a CanvasTexture
renderer.setRenderTarget(cgTarget); renderer.setClearColor(0, 0); renderer.render(virtualScene, camera)
// fragment:
// vec2 cbcr = chroma(cam); float a = smoothstep(threshold, threshold + soft, distance(cbcr, chroma(key)));
// cam.g -= max(0.0, cam.g - max(cam.r, cam.b)) * spill; // remove green spill
// gl_FragColor = vec4(mix(cg.rgb, cam, a), 1.0); // keyed actor over the virtual set
// // overlay mode instead: mix(cam, cg.rgb * exposure * whiteBalance, cg.a)