Three.js AR Depth Occlusion (Passthrough Depth Sensing)

Depth-based occlusion for passthrough AR in Three.js, where a virtual robot walks behind real furniture as every virtual fragment is tested against a real-world depth map, with soft edges, adjustable depth resolution and noise to show real-sensor artefacts.

Key topics: ar occlusion, depth sensing, webxr depth, passthrough ar, three.js shader, mixed reality

What this code does

- Inputs: a passthrough device provides colour plus a depth map of the real world (here a synthetic room rendered twice, colour and linear depth in metres).
- Per-Fragment Test: onBeforeCompile injects a lookup of real depth at gl_FragCoord and compares it with the virtual fragment's view depth.
- Soft Edges: smoothstep(−s, s, realZ − virtualZ) fades instead of cutting, hiding sensor noise at boundaries.
- Sensor Realism: low resolution (160×120) and distance-squared noise reproduce the halos and shimmering edges of real depth sensors.
- Without Occlusion: virtual content always draws over reality, the main AR immersion breaker, which is easy to see by toggling.
- Controls: occlusion, depth resolution, soft edge, depth noise and depth-map view; Enter AR requests the 'depth-sensing' feature.

Continue Learning

JavaScript Implementation

ES6+
// Depth-based occlusion: hide virtual fragments that are behind real surfaces
material.onBeforeCompile = (shader) => {
  shader.uniforms.tRealDepth = { value: depthTexture }            // metres, from the device's depth sensor
  shader.fragmentShader = shader.fragmentShader.replace('#include <dithering_fragment>', `#include <dithering_fragment>
    float realZ = texture2D(tRealDepth, gl_FragCoord.xy / resolution).r;
    gl_FragColor.a *= smoothstep(-0.04, 0.04, realZ - vViewZ);      // soft edge hides sensor noise`)
}
// WebXR: requestSession('immersive-ar', { requiredFeatures: ['depth-sensing'], depthSensing: { usagePreference: ['cpu-optimized'], dataFormatPreference: ['luminance-alpha'] } })