Reinforcement Learning

AlphaZero for 2048

AlphaZero transferred to a single-player stochastic puzzle: a residual policy/value net, chance-aware MCTS, and a 1-D engine, trained by self-play with no human gameplay heuristics.

~28%
late games reach 2048: ~20 h
on a fanless M1 MacBook: 20–51×
engine vs NumPy baseline: ~2.5M
ZeroNet parameters

Technical report

Problem

2048 has on the order of 10⁵² reachable boards and a random 2 or 4 after every move. Strong published agents still lean on hand-coded evaluation — monotonicity, smoothness, empty cells — which has to be rewritten if the rules change.

How it works

011-D engine + row-shift cache
02ZeroNet (8×128 residual)
03Chance-aware MCTS
04Batched concurrent self-play
05Replay + 8-fold D4

Store the board as a 16-element Python list with a row-shift cache — 20–51× faster than a 2-D NumPy engine, which is what lets a 20-iteration run finish in about 20 hours on a fanless M1 MacBook.
Encode tiles as one-hot powers of two into ZeroNet (8 residual blocks, 128 channels, ~2.5M parameters). Keep MCTS at inference: the network never quite matches the search-improved policy, and endgames need lookahead the policy alone cannot enumerate.
Absorb random spawns into MCTS value estimates instead of expanding chance nodes, and batch leaf evaluation across concurrent games so GPU utilisation stays above 80%. Train with 8-fold D4 augmentation, mixed precision, cosine LR, and a 10⁵ replay buffer.

From the report

Chapter 6 of the report: loss over 20 self-play iterations, 1-D engine throughput versus NumPy, and the max-tile distribution shifting right as the agent learns.

Figure 6.2 from the report: 1-D Python list engine versus a 2-D NumPy baseline, 20× to 51× faster on every operation. — Figure 6.2 — Engine throughput, log scale. 1-D list versus NumPy: 20–51× on every primitive. A full self-play step is ~480k/s versus ~21k/s.

Figure 6.3 from the report: max-tile distribution shifting right from early to late training, with 2048 in about 28% of late games. — Figure 6.3 — Max tile in early, mid, and late self-play. Late training hits 2048 in about 1 in 3 games, 4096 in about 1 in 17.

Outcome

After 20 iterations (~20 hours on a fanless M1 MacBook), late-stage self-play reaches 2048 in about 28% of games and 4096 in about 1 in 17. Early training rarely cleared 512.
Policy loss fell from ~ln 4 to ~0.5; value loss from ~0.75 to under 0.15. A full self-play step runs at ~480k/s versus ~21k/s on the NumPy baseline — overnight training instead of a long weekend.
This is not the TD + Expectimax record (32,768 in over 30% of games). The trade is a fully self-taught policy, with no human evaluation features at action selection.

Stack

Python
PyTorch
MCTS
ResNet
AMP