Three.js Reinforcement Learning (Q-Learning vs SARSA Gridworld)

Tabular reinforcement learning in Three.js, where an agent learns to reach the goal in cliff-walk, four-rooms and maze gridworlds with Q-learning or SARSA, shown as live value heights, greedy-policy arrows and the latest episode path.

Key topics: q-learning visualization, reinforcement learning, sarsa, gridworld, cliff walking, three.js ai

What this code does

- Setup: states are grid cells and actions are four moves; reward is −1 per step, −100 for a pit (back to start) and +10 at the goal.
- Q-Learning: Q(s,a) += α [r + γ maxₐ′ Q(s′,a′) − Q(s,a)], off-policy, learning the optimal greedy policy.
- SARSA: bootstraps from the action actually taken next (on-policy), so with ε-exploration it learns the safer path away from the cliff.
- Exploration: ε-greedy takes random actions with probability ε, trading exploration against exploitation.
- Visuals: tile height and colour show V(s) = maxₐ Q(s,a), arrows show the greedy action, and the yellow line is the last episode.
- Controls: environment, algorithm, α, γ, ε, steps per frame, value display and reset.

Continue Learning

JavaScript Implementation

ES6+
// Tabular Q-learning with ε-greedy exploration
const Q = new Float32Array(numStates * 4)

function step (s) {
  const a = Math.random() < epsilon ? randomAction() : argmax(Q.subarray(s * 4, s * 4 + 4))
  const { s2, reward, done } = env.step(s, a)
  const target = reward + (done ? 0 : gamma * Math.max(...Q.subarray(s2 * 4, s2 * 4 + 4)))
  Q[s * 4 + a] += alpha * (target - Q[s * 4 + a]) // temporal-difference update
  return done ? env.start : s2
}
// Visualise: tile height = max Q, arrow = argmax action