Three.js AR Face Filter (Webcam Face Tracking)

Snapchat-style AR face filters in Three.js driven by a lightweight tracker, with YCbCr skin segmentation, eye blobs for roll and yaw and dark-pixel mouth detection anchoring sunglasses, hats, cat ears and sparkles, on a synthetic subject or your webcam via getUserMedia.

Key topics: ar face filter, face tracking javascript, webcam ar, getusermedia, three.js filter, skin detection

What this code does

- Frames: a synthetic subject is rendered into a 240×180 target and read back; with the webcam, frames are mirrored into a canvas and read the same way.
- Skin Segmentation: pixels are converted to YCbCr and kept when Cr ∈ (135, 180) and Cb ∈ (80, 130), which is fairly robust to brightness.
- Face Box: the largest skin component's bounding box, clipped to 1.35 × its width so the neck doesn't stretch it.
- Eyes & Pose: dark (pupils, brows) and bright (eye whites) pixels in the 28–58% band give left and right eye centroids; roll comes from the eye line, yaw from the eye midpoint's offset and scale from eye distance.
- Expressions: dark-pixel coverage in the lower face measures mouth opening, which triggers sparkles; exponential smoothing steadies the filters.
- Controls: filter, smoothing, subject motion and debug overlays (face box, eye points, skin mask); the webcam button asks for camera permission.

Continue Learning

JavaScript Implementation

ES6+
// Lightweight face tracking for filters: skin segmentation + eye blobs, no ML
for (let i = 0; i < w * h; i++) {
  const [r, g, b] = rgba(i)
  const cr = 128 + 0.5 * r - 0.4187 * g - 0.0813 * b, cb = 128 - 0.1687 * r - 0.3313 * g + 0.5 * b
  skin[i] = cr > 135 && cr < 180 && cb > 80 && cb < 130
}
const face = boundingBox(largestComponent(skin))
const [left, right] = darkCentroids(face.upperBand)             // pupils & brows
glasses.position.set(...toNdc(mid(left, right)))
glasses.rotation.z = Math.atan2(right.y - left.y, right.x - left.x)
glasses.scale.setScalar(distance(left, right))