Three.js Reinforcement Learning (Q-Learning vs SARSA Gridworld)
Tabular reinforcement learning in Three.js, where an agent learns to reach the goal in cliff-walk, four-rooms and maze gridworlds with Q-learning or SARSA, shown as live value heights, greedy-policy arrows and the latest episode path.
Key topics: q-learning visualization, reinforcement learning, sarsa, gridworld, cliff walking, three.js ai
Game Over
What this code does
- Setup: states are grid cells and actions are four moves; reward is −1 per step, −100 for a pit (back to start) and +10 at the goal.
- Q-Learning: Q(s,a) += α [r + γ maxₐ′ Q(s′,a′) − Q(s,a)], off-policy, learning the optimal greedy policy.
- SARSA: bootstraps from the action actually taken next (on-policy), so with ε-exploration it learns the safer path away from the cliff.
- Exploration: ε-greedy takes random actions with probability ε, trading exploration against exploitation.
- Visuals: tile height and colour show V(s) = maxₐ Q(s,a), arrows show the greedy action, and the yellow line is the last episode.
- Controls: environment, algorithm, α, γ, ε, steps per frame, value display and reset.
Related Demos
Continue Learning
JavaScript Implementation
// Tabular Q-learning with ε-greedy exploration
const Q = new Float32Array(numStates * 4)
function step (s) {
const a = Math.random() < epsilon ? randomAction() : argmax(Q.subarray(s * 4, s * 4 + 4))
const { s2, reward, done } = env.step(s, a)
const target = reward + (done ? 0 : gamma * Math.max(...Q.subarray(s2 * 4, s2 * 4 + 4)))
Q[s * 4 + a] += alpha * (target - Q[s * 4 + a]) // temporal-difference update
return done ? env.start : s2
}
// Visualise: tile height = max Q, arrow = argmax action