I was always interested in the Lunar Lander game and suddenly I heard about model predictive control and quadratic programming. I want to see if the Lunar Lander RL can be adapted to that one in here.
That question led me to build three related experiments around Gymnasium's
LunarLander-v3:
- Lunar RL: a causal-transformer PPO policy trained through reinforcement learning.
- Lunar MPC: adaptive model predictive control using discrete beam search.
- Lunar QP: model predictive control whose plan is solved as a convex quadratic program.
All three control the same lander, but they answer very different questions. RL asks, “Can experience produce a policy that maps observations directly to actions?” MPC asks, “Given an approximate physics model, which short action sequence looks best from the current state?” QP asks, “Can that short-horizon plan be expressed as a fast convex optimization problem with explicit constraints?”
The progression is not a conversion of one trained checkpoint into another controller. It is an adaptation of the problem: the observation interface, action space, randomized starts, disturbance scenarios, evaluation rules, and replay tools carry forward, while the decision-making mechanism changes completely.
One lander, three ways to decide
LunarLander exposes an eight-dimensional state: horizontal and vertical position, horizontal and vertical velocity, angle, angular velocity, and two leg-contact flags. At every simulator step, a controller chooses one of four legal actions: coast, fire the left orientation engine, fire the main engine, or fire the right orientation engine.
| Question | Lunar RL | Lunar MPC | Lunar QP |
|---|---|---|---|
| Decision | One policy inference | Beam search over discrete action sequences | Convex optimization of fractional engine duties |
| Knowledge | Learned from simulated experience | Explicit approximate dynamics | Local affine dynamics and linear constraints |
| Adaptation | Requires learning or retraining | Replans and estimates engine response in flight | Re-solves and estimates engine response in flight |
| Action handling | Outputs one legal discrete action | Searches legal discrete actions | Allocates fractional requests into discrete pulses |
| Primary cost | Expensive training, cheap inference | Online combinatorial search | Online numerical optimization |
1. Lunar RL: learn the behavior before flight
Lunar RL uses Proximal Policy Optimization (PPO). During training, many simulated landers collect trajectories. PPO adjusts an actor that selects actions and a critic that estimates future return. After training, control is simple: encode the latest context, evaluate the policy once, and execute the selected discrete action.
The recorded Lunar RL policy landing from an off-pad, tilted start.
The network can process either the normal eight-value state or rendered pixels. Each timestep becomes a token containing the current state, previous action, previous reward, and optionally an Impala-CNN image embedding. Three GTrXL blocks process a causal window of tokens. The actor emits four logits; the critic represents value with 41 two-hot bins in symlog space.
There is an honest architectural lesson in the repository: the standard eight-value observation is already Markov. A small multilayer perceptron can solve it, so a transformer is unnecessary in vector mode. Memory becomes useful in pixel mode because a single image shows pose but does not directly reveal velocity or angular rate. A sequence of frames lets attention recover part of that hidden state.
The advantage of RL is that the policy can absorb behavior that is difficult to write as equations. The cost is paid during training, and the result is tied to the distribution it experienced. A center-trained checkpoint landed only six of eight randomized off-pad approaches. Training with the same position and tilt randomization produced a robust checkpoint that landed all eight in that experiment.
The repository reports a historical mean return of +282.6 for its centered checkpoint and documents stronger robust-policy evaluations. It also states an important current limitation: the shipped metrics predate a correction that made the PPO update context exactly match the context used while acting. The first retrained candidate after that correction failed the strict promotion checks, so the historical checkpoint remains in place while a corrected baseline is validated. That kind of disclosure matters more than a single high score.
2. Lunar MPC: predict, act once, and predict again
Lunar MPC moves the computation from training time to control time. It starts with an approximate six-state physics model for position, velocity, angle, and angular velocity. At each observation, it searches possible future action sequences, chooses the lowest-cost plan, applies only its first action, observes what actually happened, and repeats.
observe current state
update the engine model
predict candidate action sequences
keep the best beam
execute the first action
repeat from the next observationThis repeated loop is the defining idea of model predictive control. A prediction does not have to remain accurate for the entire landing. It needs to be useful long enough to choose the next action, because the controller replans after 0.02 simulated seconds. The default horizon contains 16 blocks of four physics steps, or 1.28 seconds of foresight, with a beam width of 32.
Adaptive MPC replans after an unannounced 30% main-engine power loss.
Exhaustively expanding four actions for 16 blocks would require examining
4**16 sequences. Beam search avoids that explosion. At each depth, it
expands the current candidates with all four actions and keeps only the 32 plans with
the lowest accumulated cost. The cost penalizes horizontal error, velocity error,
vertical descent error, tilt, angular motion, unsafe approach geometry, and excessive
main-engine use.
The model also learns during the flight. Bounded recursive least squares compares the predicted velocity change with the observed change and updates estimates of main-engine acceleration, side-engine acceleration, translational bias, angular response, and angular bias. Contact impulses and extreme accelerations are excluded because they are poor evidence of airborne engine performance.
This creates two forms of adaptation. Replanning reacts to state error even when the model remains fixed. Parameter estimation tries to correct the model itself. The distinction is visible in the fault experiment: immediately before action 100, the evaluator reduces main-engine power by 30%. The controller is not told the fault time or the new multiplier; it sees only the resulting motion.
On historical evaluation seeds 200–249, adaptive MPC passed 49 of 50 strict landing checks both normally and under the fault. The fixed-model controller also passed 49 of 50 normally but fell to 42 of 50 under the fault. This is evidence that online identification helped in that specific experiment, while still leaving a known horizontal-recovery failure and no safety certificate.
3. Lunar QP: make the plan a convex optimization
Lunar QP is still MPC. The change is the planner inside the repeated loop. Instead of beam-searching discrete sequences, it linearizes the approximate dynamics around the current state and asks OSQP to choose fractional engine duties over the horizon.
minimize 1/2 z' P z + q' z
subject to lower <= A z <= upper
The predicted dynamics have the local affine form
x[k+1] = Adx[k] + Bdu[k] + c. The implementation
condenses the state sequence into X = base + response*U, leaving the
engine requests and slack variables as optimization decisions. With 16 prediction
blocks, the QP has 96 variables: 48 engine duties and 48 nonnegative slacks.
Its quadratic objective penalizes deviation from desired position, velocity, descent rate, tilt, and angular velocity. Linear constraints bound every duty between zero and one and require the three engine fractions to sum to at most one. Soft approach limits encourage clearance, bounded tilt, and remaining inside the viewport. Explicit slack variables keep a difficult state mathematically feasible while exposing how much the preferred envelope was relaxed.
QP-based MPC uses fractional plans and executes discrete engine pulses.
The most interesting mismatch is at the actuator. The Gymnasium environment accepts one discrete action, but the convex QP may request something like 60% main engine, 10% left engine, and 30% coast. A pulse-density allocator accumulates these fractions over time, executes whichever action has the largest balance, and subtracts one from that balance. Over repeated decisions, the discrete pulse counts approximate the requested fractions.
This is a convex relaxation followed by discrete allocation, not mixed-integer optimization. The predicted fractional path and the path produced by real pulses can diverge. Replanning corrects part of that error, but a solved QP is only a numerical statement about the approximate optimization problem. It is not proof of a safe landing in the nonlinear Box2D simulator.
Lunar QP uses the same style of bounded online engine estimation as Lunar MPC. On its separate evaluation seeds 300–319, both adaptive and fixed QP passed 20 of 20 normal flights. Under the 30% engine fault, adaptive QP passed 15 of 20 and fixed QP passed 16 of 20. The result does not demonstrate an adaptation advantage for QP. It also shows why successful development seeds are not enough: all four controller/scenario combinations had passed the eight development starts.
Can the RL solution be adapted to MPC and QP?
Yes, if “adapted” means preserving the experimental system and replacing the decision rule. The reusable parts are substantial:
- The same eight controller-visible observations and four legal actions.
- The same randomized off-pad and tilted starting-pose wrapper.
- The same separation between controller information and evaluator-only simulator truth.
- The same disturbance idea: reduce engine power without telling the controller.
- Strict landing checks based on termination, settling, leg contact, and foot position.
- Recorded trajectories and self-contained browser replays for inspecting failures.
What does not transfer is the trained policy itself. PPO weights encode a nonlinear mapping learned from data; an MPC controller needs explicit transition equations and a cost; a QP additionally needs dynamics and constraints that remain affine or quadratic after local approximation. Trying to reinterpret neural-network weights as those matrices would solve a different and much harder identification problem.
A future hybrid could combine them more directly. RL could learn a residual correction to the MPC physics model, tune terminal costs, propose a warm-start plan, or provide a recovery policy when the optimizer fails. MPC could supervise safer exploration or generate demonstrations for an RL policy. Those designs require new matched experiments; none of the three current repositories claims to implement them.
Reading the results without fooling ourselves
The reported numbers are evidence about each repository, not a leaderboard. Lunar RL, Lunar MPC, and Lunar QP use different training histories, seed ranges, costs, horizons, discretization choices, and evaluation sample sizes. The MPC and QP controllers even optimize different cost functions. Comparing their pass counts directly would not isolate whether policy learning, beam search, or QP is better.
A fair head-to-head study would freeze the controller-visible state, starting poses, terrain seeds, engine faults, landing definition, time limit, compute budget, and reporting format. It would measure pass rate, worst-case behavior, return distribution, decision latency, deadline misses, and sensitivity to model error. For QP, it should also report slack and the error introduced by fractional-to-discrete allocation. For RL, it should report variation across training seeds and hardware backends.
Run the three experiments
Each repository is standalone and uses uv.
Reinforcement learning
git clone https://github.com/mraad/lunar-rl.git
cd lunar-rl
uv sync --locked
uv run lunar-rl-view --ckpt lunar_agent_robust.pt --seed 0 --greedyAdaptive beam-search MPC
git clone https://github.com/mraad/lunar-mpc.git
cd lunar-mpc
uv sync --locked
uv run lunar-mpc --episodes 8 --seed 0 --thrust-scale 0.7 \
--out dist/fault-adaptive.json --replay dist/fault-adaptive.htmlQP-based MPC
git clone https://github.com/mraad/lunar-qp.git
cd lunar-qp
uv sync --locked
uv run lunar-qp --episodes 8 --seed 0 --thrust-scale 0.7 \
--out dist/qp/fault-adaptive.json --replay dist/qp/fault-adaptive.htmlThe generated MPC and QP replay pages expose more than the final score. They show the actual trajectory, predicted path, chosen plan, engine estimates, and decision timing. The QP replay adds fractional requests, executed pulses, solver status, residuals, and slack. The RL viewer shows policy probabilities, critic values, and attention over the recent context. Looking inside a failed landing is often more useful than adding another decimal place to an average.
What the three controllers taught me
RL can discover a capable policy without a hand-written flight model, but its behavior depends on training coverage and careful validation. MPC makes assumptions visible and can react immediately when observed motion differs from prediction, but it pays for planning at every step. QP makes objectives and constraints especially inspectable and solves the relaxed plan efficiently, while introducing local-linearization and actuator- allocation errors.
The most useful outcome of adapting LunarLander across these methods is not choosing one universal winner. It is seeing the trade clearly: learned behavior, discrete online search, and constrained convex optimization each fail in different ways. Keeping the environment, disturbances, evidence, and replay tools consistent turns those failures into something we can study rather than hide behind a score.
No comments:
Post a Comment