LunarLander looks simple: observe eight numbers, choose one of four engine actions, and guide a small craft between two flags. In practice, it exposes several failure modes that make Deep Q-Learning interesting. My first promising agents learned to approach the landing pad, then hovered above it until the episode timed out. Other runs climbed past a score of 100 and later collapsed.
I built LunarLanderDRL to work through those problems in a compact PyTorch implementation. The final agent combines a Deep Q-Network (DQN), proportional prioritized experience replay, Double DQN, Huber loss, Polyak target updates, and three-step returns. Across three training seeds, the resulting checkpoints averaged 265.7 on 90 held-out episodes and recorded zero crashes.
Two consecutive rollouts of the shipped policy, scoring 308.1 and 272.1.
The problem
Gymnasium's LunarLander-v3 returns an eight-dimensional observation:
horizontal and vertical position, horizontal and vertical velocity, angle, angular
velocity, and two leg-contact flags. The discrete action space contains four choices:
do nothing, fire the left orientation engine, fire the main engine, or fire the right
orientation engine.
A DQN approximates the action-value function Q(s, a). Given a state, the
network emits one value for each action, and a greedy policy selects the largest one.
The model in this project is deliberately small:
8 inputs -> 256 -> 128 -> 64 -> 4 Q-valuesThe hidden layers use ReLU activations and Xavier initialization. During training, an epsilon-greedy policy sometimes takes a random action, with epsilon decaying from exploration toward exploitation. The network architecture is the easy part. Stable learning depends on what transitions we replay, how we calculate targets, and how we decide that one checkpoint is better than another.
Prioritized experience replay
Uniform replay treats every stored transition as equally informative. Prioritized experience replay (PER) focuses training on transitions where the current prediction is most wrong. This implementation assigns each transition the priority
p(i) = (|TD error(i)| + epsilon) ** alpha
alpha controls how strongly priority affects sampling. A value of zero reduces
PER to uniform replay; the default is 0.6. A new transition has no TD error yet, so it
enters with the highest priority seen so far. This ensures that it is replayed once
before being assigned a data-driven priority.
The replay buffer stores states, actions, rewards, terminal flags, next states, and
bootstrap discounts in flat NumPy arrays. Priorities live in a binary sum tree. Each
parent stores the sum of its children, which lets the agent update a priority or locate
a sampled leaf in O(log N) time. A batch is sampled from equal-mass segments
of the total priority, reducing variance compared with drawing every item independently.
Prioritized sampling changes the training distribution, so each sampled loss receives an importance-sampling correction:
w(i) = (N * P(i)) ** -betaThe weights are normalized by the maximum weight in the batch. Beta is annealed from 0.4 to 1.0, making the correction stronger as the policy approaches convergence.
Stopping Q-value overestimation from taking over
A vanilla DQN uses a maximum over estimated values when it constructs the bootstrap target. Estimation noise makes that maximum optimistic. Because the target is itself used to train future estimates, the bias can compound until a policy that appeared to be improving suddenly collapses.
Double DQN separates action selection from action evaluation. The online network picks the next action, while the target network evaluates it:
next_action = online(next_state).argmax()
next_value = target(next_state)[next_action]
target_q = reward + discount * next_valueThe target network is not copied in occasional hard jumps. It is updated with Polyak averaging on every training step:
target = (1 - tau) * target + tau * online
The default tau is 1e-3. This gives the target a slowly moving
reference point while the online network continues to learn.
The project also uses Huber loss instead of squared error. This choice matters when it is paired with PER: the replay sampler intentionally selects transitions with large TD errors, and squaring an outlier can let one sample dominate the gradient. Huber loss is quadratic close to zero and linear for large errors. Gradient norms are clipped at 10 as a final guard against unstable updates.
Why the one-step agent learned to hover
The most revealing failure was not a crash. A one-step agent often descended near the pad, stabilized, and hovered until Gymnasium's 1,000-step limit. That behavior makes sense from the agent's temporary view of the world. LunarLander provides shaping reward for remaining close to the pad at low velocity, while the large touchdown reward is still hundreds of actions away. With one-step bootstrapping, that delayed reward moves backward through the value function slowly.
Three-step returns propagate the landing signal more quickly. Instead of storing only one immediate reward, the buffer stores a folded return:
R(t) = r(t) + gamma*r(t+1) + gamma**2*r(t+2)
target_q = R(t) + gamma**k * Q(next_state, next_action)
Two implementation details are essential. First, the bootstrap multiplier must be
gamma**k, where k is the number of rewards actually folded into
that transition. The buffer therefore stores a discount beside each transition.
Reusing a single gamma would silently produce incorrect targets.
Second, the pending reward window must be flushed at every episode boundary. Otherwise,
the last n_step - 1 transitions disappear. Those are often the exact
transitions containing touchdown, so dropping them removes the examples the agent most
needs. If a real terminal occurs inside the window, the fold stops there. A time-limit
truncation ends the rollout but does not zero a state value that could continue beyond
the artificial cutoff.
Evaluate the policy you will actually deploy
Early versions saved checkpoints using the moving average of training rewards. That metric is contaminated by epsilon-greedy exploration. A lucky random action can improve a score, and a useful exploratory action can also lower it. The checkpoint should be selected using the deterministic policy that will run after training.
The training loop now evaluates a fully greedy policy every 25 episodes on a fixed set of seeds. Evaluation uses a separate Gymnasium environment, so resetting it never changes the training environment's random-number stream. This produced a striking example: one checkpoint had a training average of 239.8 but scored only 160.6 under greedy evaluation. In another run, the training average was just 78.2 while the greedy policy scored 269.1. Selecting on training reward would have preserved the first and discarded the second.
Measured results
I trained three seeds with the following configuration:
uv run main.py --n_episodes 4000 --eps_ratio 0.35 --eps_final 0.01 \
--learning_rate 5e-4 --warm_start 5000 --n_step 3 \
--eval_episodes 20 --solved_score 275The runs stopped between episodes 474 and 999 and took roughly ten minutes each on an M-series CPU. I evaluated every checkpoint on the same 30 seeds, numbered 1000 through 1029. Those held-out seeds are disjoint from the seeds used to select checkpoints.
| Training seed | Mean | Median | Solved | Crashed |
|---|---|---|---|---|
| 42 | 280.6 | 281.2 | 30/30 | 0 |
| 43 | 243.3 | 278.4 | 24/30 | 0 |
| 44 (shipped model) | 273.1 | 286.5 | 28/30 | 0 |
| Pooled | 265.7 | 282.1 | 82/90 | 0/90 |
The comparison that changed the design was the one-step run. With otherwise matching settings, it averaged 220.3, solved 25 of 30 held-out episodes, and crashed twice. The three-step runs produced no crashes in 90 episodes. Eight episodes finished below 200, but these were soft or slow landings scoring between 21 and 46, rather than destructive impacts.
I ship the seed-44 checkpoint even though seed 42 scored higher. Seed 44 lies closer to the pooled result, so it is a more representative artifact than the luckiest run.
Run it yourself
The project uses uv, pins dependencies in
uv.lock, and includes a trained model. Clone the repository and install the
environment:
git clone https://github.com/mraad/LunarLanderDRL.git
cd LunarLanderDRL
uv syncReplay the shipped policy into an MP4:
uv run render.py
# Scores only, without creating video
uv run render.py --no-saveTrain a new policy and monitor it with TensorBoard:
uv run main.py --n_episodes 4000 --learning_rate 5e-4 \
--warm_start 5000 --n_step 3
uv run tensorboard --logdir logsThe automated checks cover schedules, both replay buffers, the sum tree, n-step folding and flushing, and a short end-to-end training run against the real environment:
uv run pytest test_rl.pyWhat I learned
The useful lesson was not that one algorithmic trick solves LunarLander. The pieces address different sources of failure. PER spends more updates on surprising transitions. Importance weights correct its sampling bias. Double DQN reduces optimistic targets. Huber loss and gradient clipping keep high-error examples from destabilizing an update. Three-step returns move delayed touchdown information backward faster. Greedy, isolated evaluation selects the policy that will actually be used.
The project remains intentionally small enough to read end to end. The next practical improvements would be training every few environment steps instead of every step, scheduling epsilon by step rather than episode, annealing PER beta across the full run, and adding a dueling value/advantage head. The current implementation and trained weights are available under the Apache 2.0 license at github.com/mraad/LunarLanderDRL.
No comments:
Post a Comment