AlphaZero for 2048

2026 · Reinforcement Learning

Python, PyTorch, MCTS, ResNet, Mixed precision (AMP)

An AlphaZero agent that learns 2048 from self-play alone. A residual policy/value network scores boards and suggests moves; a chance-aware Monte Carlo Tree Search (MCTS) plans around the random tile spawns, on top of a 1-D board engine written for speed. After 20 iterations it reaches 2048 in about 28% of late self-play games, with no human evaluation features anywhere in move selection.

Problem

The record-holding 2048 agents lean on hand-tuned features: monotonicity, smoothness, empty cells. Change the rules and you retune them. I wanted an agent that is never told what a good board looks like. That means learning an evaluation over something like 10⁵² reachable boards, with a random 2 or 4 dropped in after every move.

Approach

The board is a 16-element Python list with a row-shift cache, not a 4×4 NumPy array. At that size NumPy's per-call overhead dominates: a clone → move → spawn → legal-moves cycle runs at ~480k/s on the list and ~21k/s on NumPy. Self-play is engine-bound, so that is training time saved directly.
Tile powers are one-hot encoded into 16 channels and fed to ZeroNet: 8 residual blocks, 128 channels, about 2.5M parameters. MCTS stays on at inference because the network never fully matches its search-improved targets, and endgames need lookahead the policy alone does not have.
Random spawns feed MCTS value estimates without expanding chance nodes. Batching leaf evaluation across concurrent games took GPU utilisation from a few percent to over 80%, the single biggest throughput win. Training used 8-fold D4 augmentation, mixed precision, a cosine learning rate from 10⁻³ to 10⁻⁵, and a 10⁵-transition replay buffer.

Results

After 20 self-play iterations, about 20 hours on one workstation with a single GPU/MPS device, late-stage self-play reaches 2048 in about 28% of games and 4096 in about 1 in 17. In the first five iterations it rarely cleared 512.
Policy cross-entropy fell from ~ln 4 to ~0.5, value MSE from ~0.75 to under 0.15, with no collapse along the way. The 1-D engine is what made an overnight run possible instead of a long weekend.

Limits

This is nowhere near the TD + Expectimax record (32,768 in over 30% of games). What it has instead is a policy taught by nothing but play.

Metrics

late games reach 2048: 28%
engine vs NumPy: 20–51×
ZeroNet parameters: ~2.5M
one workstation, single GPU/MPS: ~20 h

Architecture

1-D engine + row-shift cache
ZeroNet (8×128 residual)
Chance-aware MCTS
Batched concurrent self-play
Replay buffer + 8-fold D4

From the report

From chapter 6 of the report: loss over 20 iterations, engine throughput against NumPy, and the max-tile distribution moving right as training goes on.

Line chart: total, policy, and value loss across 20 self-play iterations, all trending down without instability. — Figure 6.1 — Training loss per iteration. Policy cross-entropy falls from ~ln 4 to ~0.5; value MSE from ~0.75 to under 0.15.

Log-scale bar chart comparing the 1-D Python list engine against a 2-D NumPy baseline across engine primitives. — Figure 6.2 — Engine throughput, log scale. 1-D list versus NumPy: 20–51× on every primitive. A clone → move → spawn → legal-moves cycle runs at ~480k/s versus ~21k/s.

Grouped bar chart of the maximum tile reached at game end across early, middle, and late training stages, shifting right over training. — Figure 6.3 — Max tile at game end, by training stage. Late training ends on exactly 2048 in about 28% of games and on 4096 in about 1 in 17.