Teaching a Lunar Lander to Stop Hovering: DQN, Prioritized Replay, and n-Step Returns
LunarLander looks simple: observe eight numbers, choose one of four engine actions, and guide a small craft between two flags. In practice, it exposes several failure modes that make Deep Q-Learning interesting. My first promising agents learned to approach the landing pad, then hovered above it until the episode timed out. Other runs climbed past a score of 100 and later collapsed. I built LunarLanderDRL to work through those problems in a compact PyTorch implementation. The final agent combines a Deep Q-Network (DQN), proportional prioritized experience replay, Double DQN, Huber loss, Polyak target updates, and three-step returns. Across three training seeds, the resulting checkpoints averaged 265.7 on 90 held-out episodes and recorded zero crashes . Two consecutive rollouts of the shipped policy, scoring 308.1 and 272.1. The problem Gymnasium's LunarLander-v3 returns an eight-dimensional observation: horizontal and vertical position, ...