Three.js Mixed Reality Compositing (Light Estimation & Shadows)
Make virtual objects belong in a real scene in Three.js, with a split view comparing a naive paste against light estimation from camera pixels, a ShadowMaterial shadow catcher, an environment probe for reflections and matched sensor grain, plus WebXR immersive-ar where supported.
Key topics: mixed reality, light estimation, shadow catcher, ar compositing, environment probe, webxr ar
Game Over
What this code does
- Camera Feed: a synthetic living room with window light is rendered into a target and shown full-screen as the "passthrough" image.
- Light Estimation: the mean pixel colour drives hemisphere fill, and the brightest image region (the window) is unprojected into a key-light direction and intensity.
- Contact Shadows: an invisible floor with ShadowMaterial shows only the shadows virtual objects cast, the most important grounding cue.
- Reflections: a CubeCamera in the real room acts as an environment probe (as ARKit and ARCore provide), so the chrome ball reflects the window and sofa.
- Sensor Matching: film grain over the composite keeps virtual edges from looking too clean.
- Controls: split position, each integration step, and real-world lighting presets; Enter AR requests immersive-ar with light estimation.
Related Demos
Continue Learning
JavaScript Implementation
// Make virtual objects belong in a real scene
const shadowCatcher = new THREE.Mesh(floorPlane, new THREE.ShadowMaterial({ opacity: 0.45 })) // only shadows show
shadowCatcher.receiveShadow = true; virtualObject.castShadow = true
// Light estimation from camera pixels: mean colour → ambient, brightest region → key-light direction
hemi.color.copy(meanColour); sun.position.copy(directionToBrightestRegion).multiplyScalar(4)
// Environment probe → reflections (ARKit/ARCore provide one; here a CubeCamera captures the room)
cubeCamera.update(renderer, realScene); virtualScene.environment = cubeRenderTarget.texture
// In a real AR session, request 'light-estimation' and use XRLightProbe / XREstimatedLight