Three.js Word Embeddings & Attention (NLP Visualization)
Explore a 21-dimensional word-embedding space in Three.js, projected to 3D with PCA, solving analogies such as king − man + woman ≈ queen with arrows, and showing scaled dot-product self-attention between the words of a sentence.
Key topics: word embeddings visualization, word2vec analogy, nlp visualization, self attention, pca 3d, three.js
Game Over
What this code does
- Embeddings: each word is a vector; here dimensions encode features such as person, gender, royalty, country and polarity, plus noise, a stand-in for learned word2vec/GloVe vectors.
- PCA: power iteration on the covariance matrix finds the three directions of greatest variance used for plotting.
- Analogies: relations become consistent offsets, so c − a + b lands near the answer (shown as a white wireframe), ranked by cosine similarity excluding the inputs.
- Self-Attention: each token's weights are softmax(q·k / √d) over the sentence, drawn as arcs whose brightness is the weight, cycling through tokens.
- Noise: raising it shows how analogies degrade as the structure blurs.
- Controls: analogy, sentence, attention toggle, embedding noise and sway.
Related Demos
Continue Learning
JavaScript Implementation
// Word vectors: similar meanings → nearby vectors; relations → consistent offsets
const cosine = (a, b) => dot(a, b) / Math.sqrt(dot(a, a) * dot(b, b))
const query = add(sub(vec.king, vec.man), vec.woman) // king − man + woman
const best = words.filter(w => !['king', 'man', 'woman'].includes(w))
.sort((x, y) => cosine(query, vec[y]) - cosine(query, vec[x]))[0] // → "queen"
// Self-attention weights for token i: softmax over j of (q_i · k_j) / √d
const weights = softmax(tokens.map(t => dot(vec[tokens[i]], vec[t]) / Math.sqrt(d)))
// Display: PCA → 3D positions, THREE.Sprite labels, ArrowHelper for the analogy offsets