I was always interested in the Lunar Lander game and suddenly I heard about model predictive control and quadratic programming. I want to see if the Lunar Lander RL can be adapted to that one in here.
That question led me to build three related experiments around Gymnasium's
LunarLander-v3:
- Lunar RL: a causal-transformer PPO policy trained through reinforcement learning.
- Lunar MPC: adaptive model predictive control using discrete beam search.
- Lunar QP: model predictive control whose plan is solved as a convex quadratic program.
All three control the same lander, but they answer very different questions. RL asks,
“Can experience produce a policy that maps observations directly to actions?” MPC asks,
“Given an approximate physics model, which short action sequence looks best from the
current state?” QP asks, “Can that short-horizon plan be expressed as a fast convex
optimization problem with explicit constraints?”
The progression is not a conversion of one trained checkpoint into another controller.
It is an adaptation of the problem: the observation interface, action space,
randomized starts, disturbance scenarios, evaluation rules, and replay tools carry
forward, while the decision-making mechanism changes completely.
One lander, three ways to decide
LunarLander exposes an eight-dimensional state: horizontal and vertical position,
horizontal and vertical velocity, angle, angular velocity, and two leg-contact flags.
At every simulator step, a controller chooses one of four legal actions: coast, fire
the left orientation engine, fire the main engine, or fire the right orientation
engine.
| Question |
Lunar RL |
Lunar MPC |
Lunar QP |
| Decision |
One policy inference |
Beam search over discrete action sequences |
Convex optimization of fractional engine duties |
| Knowledge |
Learned from simulated experience |
Explicit approximate dynamics |
Local affine dynamics and linear constraints |
| Adaptation |
Requires learning or retraining |
Replans and estimates engine response in flight |
Re-solves and estimates engine response in flight |
| Action handling |
Outputs one legal discrete action |
Searches legal discrete actions |
Allocates fractional requests into discrete pulses |
| Primary cost |
Expensive training, cheap inference |
Online combinatorial search |
Online numerical optimization |
1. Lunar RL: learn the behavior before flight
Lunar RL uses Proximal Policy
Optimization (PPO). During training, many simulated landers collect trajectories. PPO
adjusts an actor that selects actions and a critic that estimates future return. After
training, control is simple: encode the latest context, evaluate the policy once, and
execute the selected discrete action.
The recorded Lunar RL policy landing from an off-pad, tilted start.
The network can process either the normal eight-value state or rendered pixels. Each
timestep becomes a token containing the current state, previous action, previous
reward, and optionally an Impala-CNN image embedding. Three GTrXL blocks process a
causal window of tokens. The actor emits four logits; the critic represents value with
41 two-hot bins in symlog space.
There is an honest architectural lesson in the repository: the standard eight-value
observation is already Markov. A small multilayer perceptron can solve it, so a
transformer is unnecessary in vector mode. Memory becomes useful in pixel mode because
a single image shows pose but does not directly reveal velocity or angular rate. A
sequence of frames lets attention recover part of that hidden state.
The advantage of RL is that the policy can absorb behavior that is difficult to write
as equations. The cost is paid during training, and the result is tied to the
distribution it experienced. A center-trained checkpoint landed only six of eight
randomized off-pad approaches. Training with the same position and tilt randomization
produced a robust checkpoint that landed all eight in that experiment.
The repository reports a historical mean return of +282.6 for its centered checkpoint
and documents stronger robust-policy evaluations. It also states an important current
limitation: the shipped metrics predate a correction that made the PPO update context
exactly match the context used while acting. The first retrained candidate after that
correction failed the strict promotion checks, so the historical checkpoint remains in
place while a corrected baseline is validated. That kind of disclosure matters more
than a single high score.
2. Lunar MPC: predict, act once, and predict again
Lunar MPC moves the computation
from training time to control time. It starts with an approximate six-state physics
model for position, velocity, angle, and angular velocity. At each observation, it
searches possible future action sequences, chooses the lowest-cost plan, applies only
its first action, observes what actually happened, and repeats.
observe current state
update the engine model
predict candidate action sequences
keep the best beam
execute the first action
repeat from the next observation
This repeated loop is the defining idea of model predictive control. A prediction does
not have to remain accurate for the entire landing. It needs to be useful long enough
to choose the next action, because the controller replans after 0.02 simulated seconds.
The default horizon contains 16 blocks of four physics steps, or 1.28 seconds of
foresight, with a beam width of 32.
Adaptive MPC replans after an unannounced 30% main-engine power loss.
Exhaustively expanding four actions for 16 blocks would require examining
4**16 sequences. Beam search avoids that explosion. At each depth, it
expands the current candidates with all four actions and keeps only the 32 plans with
the lowest accumulated cost. The cost penalizes horizontal error, velocity error,
vertical descent error, tilt, angular motion, unsafe approach geometry, and excessive
main-engine use.
The model also learns during the flight. Bounded recursive least squares compares the
predicted velocity change with the observed change and updates estimates of main-engine
acceleration, side-engine acceleration, translational bias, angular response, and
angular bias. Contact impulses and extreme accelerations are excluded because they are
poor evidence of airborne engine performance.
This creates two forms of adaptation. Replanning reacts to state error even when the
model remains fixed. Parameter estimation tries to correct the model itself. The
distinction is visible in the fault experiment: immediately before action 100, the
evaluator reduces main-engine power by 30%. The controller is not told the fault time
or the new multiplier; it sees only the resulting motion.
On historical evaluation seeds 200–249, adaptive MPC passed 49 of 50 strict landing
checks both normally and under the fault. The fixed-model controller also passed 49 of
50 normally but fell to 42 of 50 under the fault. This is evidence that online
identification helped in that specific experiment, while still leaving a known
horizontal-recovery failure and no safety certificate.
3. Lunar QP: make the plan a convex optimization
Lunar QP is still MPC. The change
is the planner inside the repeated loop. Instead of beam-searching discrete sequences,
it linearizes the approximate dynamics around the current state and asks OSQP to choose
fractional engine duties over the horizon.
minimize 1/2 z' P z + q' z
subject to lower <= A z <= upper
The predicted dynamics have the local affine form
x[k+1] = Adx[k] + Bdu[k] + c. The implementation
condenses the state sequence into X = base + response*U, leaving the
engine requests and slack variables as optimization decisions. With 16 prediction
blocks, the QP has 96 variables: 48 engine duties and 48 nonnegative slacks.
Its quadratic objective penalizes deviation from desired position, velocity, descent
rate, tilt, and angular velocity. Linear constraints bound every duty between zero and
one and require the three engine fractions to sum to at most one. Soft approach limits
encourage clearance, bounded tilt, and remaining inside the viewport. Explicit slack
variables keep a difficult state mathematically feasible while exposing how much the
preferred envelope was relaxed.
QP-based MPC uses fractional plans and executes discrete engine pulses.
The most interesting mismatch is at the actuator. The Gymnasium environment accepts
one discrete action, but the convex QP may request something like 60% main engine, 10%
left engine, and 30% coast. A pulse-density allocator accumulates these fractions over
time, executes whichever action has the largest balance, and subtracts one from that
balance. Over repeated decisions, the discrete pulse counts approximate the requested
fractions.
This is a convex relaxation followed by discrete allocation, not mixed-integer
optimization. The predicted fractional path and the path produced by real pulses can
diverge. Replanning corrects part of that error, but a solved QP is only a numerical
statement about the approximate optimization problem. It is not proof of a safe
landing in the nonlinear Box2D simulator.
Lunar QP uses the same style of bounded online engine estimation as Lunar MPC. On its
separate evaluation seeds 300–319, both adaptive and fixed QP passed 20 of 20 normal
flights. Under the 30% engine fault, adaptive QP passed 15 of 20 and fixed QP passed 16
of 20. The result does not demonstrate an adaptation advantage for QP. It also
shows why successful development seeds are not enough: all four controller/scenario
combinations had passed the eight development starts.
Can the RL solution be adapted to MPC and QP?
Yes, if “adapted” means preserving the experimental system and replacing the decision
rule. The reusable parts are substantial:
- The same eight controller-visible observations and four legal actions.
- The same randomized off-pad and tilted starting-pose wrapper.
- The same separation between controller information and evaluator-only simulator truth.
- The same disturbance idea: reduce engine power without telling the controller.
- Strict landing checks based on termination, settling, leg contact, and foot position.
- Recorded trajectories and self-contained browser replays for inspecting failures.
What does not transfer is the trained policy itself. PPO weights encode a nonlinear
mapping learned from data; an MPC controller needs explicit transition equations and a
cost; a QP additionally needs dynamics and constraints that remain affine or quadratic
after local approximation. Trying to reinterpret neural-network weights as those
matrices would solve a different and much harder identification problem.
A future hybrid could combine them more directly. RL could learn a residual correction
to the MPC physics model, tune terminal costs, propose a warm-start plan, or provide a
recovery policy when the optimizer fails. MPC could supervise safer exploration or
generate demonstrations for an RL policy. Those designs require new matched
experiments; none of the three current repositories claims to implement them.
Reading the results without fooling ourselves
The reported numbers are evidence about each repository, not a leaderboard. Lunar RL,
Lunar MPC, and Lunar QP use different training histories, seed ranges, costs, horizons,
discretization choices, and evaluation sample sizes. The MPC and QP controllers even
optimize different cost functions. Comparing their pass counts directly would not
isolate whether policy learning, beam search, or QP is better.
A fair head-to-head study would freeze the controller-visible state, starting poses,
terrain seeds, engine faults, landing definition, time limit, compute budget, and
reporting format. It would measure pass rate, worst-case behavior, return distribution,
decision latency, deadline misses, and sensitivity to model error. For QP, it should
also report slack and the error introduced by fractional-to-discrete allocation. For
RL, it should report variation across training seeds and hardware backends.
Run the three experiments
Each repository is standalone and uses uv.
Reinforcement learning
git clone https://github.com/mraad/lunar-rl.git
cd lunar-rl
uv sync --locked
uv run lunar-rl-view --ckpt lunar_agent_robust.pt --seed 0 --greedy
Adaptive beam-search MPC
git clone https://github.com/mraad/lunar-mpc.git
cd lunar-mpc
uv sync --locked
uv run lunar-mpc --episodes 8 --seed 0 --thrust-scale 0.7 \
--out dist/fault-adaptive.json --replay dist/fault-adaptive.html
QP-based MPC
git clone https://github.com/mraad/lunar-qp.git
cd lunar-qp
uv sync --locked
uv run lunar-qp --episodes 8 --seed 0 --thrust-scale 0.7 \
--out dist/qp/fault-adaptive.json --replay dist/qp/fault-adaptive.html
The generated MPC and QP replay pages expose more than the final score. They show the
actual trajectory, predicted path, chosen plan, engine estimates, and decision timing.
The QP replay adds fractional requests, executed pulses, solver status, residuals, and
slack. The RL viewer shows policy probabilities, critic values, and attention over the
recent context. Looking inside a failed landing is often more useful than adding another
decimal place to an average.
What the three controllers taught me
RL can discover a capable policy without a hand-written flight model, but its behavior
depends on training coverage and careful validation. MPC makes assumptions visible and
can react immediately when observed motion differs from prediction, but it pays for
planning at every step. QP makes objectives and constraints especially inspectable and
solves the relaxed plan efficiently, while introducing local-linearization and actuator-
allocation errors.
The most useful outcome of adapting LunarLander across these methods is not choosing one
universal winner. It is seeing the trade clearly: learned behavior, discrete online
search, and constrained convex optimization each fail in different ways. Keeping the
environment, disturbances, evidence, and replay tools consistent turns those failures
into something we can study rather than hide behind a score.
References
