Three.js WebXR Hand Tracking (Gesture Recognition)

WebXR hand tracking with gesture recognition in Three.js, classifying open palm, fist, point, peace, thumbs up, rock and pinch from 25-joint hand skeletons measured live, using real XRHand joints in VR or an animated procedural hand on desktop.

Key topics: webxr hand tracking, gesture recognition, xrhand, three.js vr, hand joints, pinch detection

What this code does

- XRHand: requesting the 'hand-tracking' feature gives 25 joint poses per hand each frame, via renderer.xr.getHand(i).joints in three.js.
- Finger Curl: the bend angles between successive bone segments are summed along each finger and normalised to 0 (straight) … 1 (curled).
- Templates: each gesture is a curl vector plus a pinch flag (thumb tip to index tip under 35% of hand size); the nearest template wins.
- Stability: a 12-frame majority vote removes flicker at transitions, which matters with noisy real tracking.
- Desktop Simulation: a procedural hand blends between templates with tracking noise, and the classifier only sees measured joint positions, exactly as in VR.
- Controls: cycle gestures, hold one gesture, seconds per gesture and tracking noise; the Enter VR button appears when the browser supports it.

Continue Learning

JavaScript Implementation

ES6+
// WebXR hand tracking: request the feature, then read 25 joint poses per hand every frame
const session = await navigator.xr.requestSession('immersive-vr', { optionalFeatures: ['hand-tracking'] })
renderer.xr.setSession(session)
const hand = renderer.xr.getHand(0); scene.add(hand)       // hand.joints['index-finger-tip'] etc.

function curl (ids) {                                       // summed bend angle along a finger
  let total = 0
  for (let k = 1; k < ids.length - 1; k++) total += angleBetween(joint(ids[k]).sub(joint(ids[k - 1])), joint(ids[k + 1]).sub(joint(ids[k])))
  return total / 3.2
}
const pinching = joint('thumb-tip').distanceTo(joint('index-finger-tip')) < 0.02