Three.js Computer Vision Object Detection Overlay
A real-time detection pipeline on a Three.js scene, where frames from an offscreen render-target "sensor" are read back, colour-segmented in HSV and grouped with union-find connected components into labelled bounding boxes drawn as an overlay.
Key topics: object detection overlay, computer vision javascript, connected components, color segmentation, readrenderTargetPixels, three.js
Game Over
What this code does
- Sensor: the scene is rendered into a 192×128 `WebGLRenderTarget` and copied to the CPU with `readRenderTargetPixels`, like a camera frame.
- Segmentation: each pixel is converted to HSV and kept when saturated enough, then snapped to the nearest class hue (red, yellow, green, blue).
- Connected Components: union-find over 4-neighbours merges same-class pixels into blobs, and each blob's min/max gives its bounding box.
- Filtering: blobs below the minimum area are dropped, and the label percentage is how much of its box the blob fills.
- Overlay: boxes and labels are drawn in normalised device coordinates with an orthographic camera after the main pass.
- Controls: view (boxes or mask), minimum blob area, saturation threshold, mask inset and object motion.
Related Demos
Continue Learning
JavaScript Implementation
// Read the rendered frame back and find coloured blobs → bounding boxes
renderer.setRenderTarget(sensor); renderer.render(scene, camera); renderer.setRenderTarget(null)
renderer.readRenderTargetPixels(sensor, 0, 0, w, h, pixels)
const cls = classifyByHue(pixels) // -1 = background, else class index
const parent = Int32Array.from({ length: w * h }, (_, i) => i)
const find = (a) => { while (parent[a] !== a) a = parent[a] = parent[parent[a]]; return a }
for (let i = 0; i < w * h; i++) {
if (cls[i] < 0) continue
if (i % w && cls[i - 1] === cls[i]) parent[find(i)] = find(i - 1)
if (i >= w && cls[i - w] === cls[i]) parent[find(i)] = find(i - w)
}
// grow a min/max box per root, drop tiny ones, draw them in an orthographic overlay