The Trainee Got Better. It Still Cannot Fly Alone.
I ended The Engineer and the Trainee with a plan: roll out the shielded pilot, relabel every visited state with MPC, retrain, and see whether the trainee could finally fly on its own. That is still the plan. But before getting there I did something lazier, and the result was more interesting than I expected. I also fixed something that had been bothering me every time I watched a flight.
The same recorded flight as last time, re-run with a smoother controller and a retrained model. The engine loses 60% of its thrust at ten seconds, the estimate falls to 40%, and the shield now steps in on about one command in eight instead of one in three.
The lander had the shakes
Watching a descent, the ship twitched. Not a rendering artifact — the controller really was changing its mind. I measured it: over 30 flights the beam search picked a different command on 77.7% of decisions. Every fifth of a second it re-derived the whole plan from scratch, and when two commands were nearly tied it would happily alternate between them. The tank chattered between throttle settings and the attitude jets flipped sign.
The fix is one term in the cost function: charge a candidate for differing from the command actually being flown. One knob, normalized so it covers tilt and throttle together. Here is what it bought, measured over seeds 3000–3009 on all three pads:
| No penalty | With penalty | |
|---|---|---|
| Decisions that change command (nominal) | 77.7% | 20.6% |
| Stage-to-stage tilt jerk | 5.37° RMS | 1.69° RMS |
| Landings, all three scenarios | 30/30 | 30/30 |
| Fuel remaining at ×0.4 thrust | 62.6 | 65.6 |
Not a trade at all, which surprised me. Smoother, and it landed the same flights using less fuel and slightly fewer decisions. The commands it stopped issuing were mostly ones it would have undone on the next stage anyway.
The same fix broke the sibling project
Naturally I tried the identical change in lunar-mpc, the Box2D controller this one descends from. On nominal flights it looked like a win: 49 of 50 strict landing checks went to 50 of 50. I nearly kept it.
Then I measured the engine-fault scenario — the one that repository exists to demonstrate — and it fell from 49/50 to 41/50. The fixed-model controller went from 42/50 to 24/50. The reason is obvious in hindsight: that lander's main engine is binary, on or off, so rapid pulsing is precisely how it holds a thrust level between the two. Charging for the pulses removes its only means of modulating thrust. Here, with three throttle settings, holding one is a real option.
So I reverted it there and wrote the numbers into that repository's research limits, because a rule that only holds in one of two codebases is worth recording where someone will trip over it. The broader lesson I have now written into my own notes: sweep a control-cost change over every published scenario before adopting it, not the one that runs quickest.
A consequence I did not plan for
Once the controller penalizes changing command, its next choice depends on what it is currently doing. Roughly four decisions in five are now “hold what you are doing.” That makes the flown command the single most predictive feature of the teacher's answer — and the trainee could not see it.
So the prompt gained one sentence: “Current command: tilt hold, engine half.” I want to be precise about why that is not a relapse into the hint-giving of the first experiment. MPC's requested command is the teacher's answer, and the earlier post showed Laya simply copies it. The flown command is actuator state, the kind any onboard controller reads from its own vehicle. The trainee still has to decide what to do next.
Retraining, on four GPUs this time
Same recipe as before, except the data came from the smoothed teacher and the run used all four RTX PRO 4500 Blackwell cards instead of three. Eight epochs in 235 seconds. Per-question validation accuracy went from 80.0% to 90.2%, and it passed 85% after the first epoch, better than anything the previous run reached in eight. A genuinely better student.
Then I flew it on the same 90 held-out flights, and this is the part worth the post:
| Previous checkpoint | Retrained | |
|---|---|---|
| Validation accuracy | 80.0% | 90.2% |
| Landings, unshielded | 0 / 90 | 1 / 90 |
| Landings, with the MPC shield | 90 / 90 | 90 / 90 |
| Laya's own choice kept | 55–62% | 80–87% |
| Shield overrides | 30–37% | 12–17% |
Ten points of accuracy bought exactly one landing out of ninety. The trainee is still not a pilot. One wrong command puts the ship slightly off the teacher's trajectory, the next state is less familiar, agreement collapses to about 17%, and the flight diverges. That is behaviour cloning without a corrective signal, and making the student more accurate on the teacher's own path does not fix it — which is a sharper argument for DAgger than I had before, because it says the missing signal is off-trajectory states rather than label quality.
What the better student did buy is a shield that works less. Overrides roughly halved. Four commands in five that fly the ship are now the model's own. The fused pilot leans further toward the trainee while physics still holds the veto.
About Jev, since everyone is asking
TypeSafe AI put Jev into limited early access on 15 September, and the reaction has been loud. It is not a language model: you hand it a block of state and typed questions, and it returns a probability distribution per question plus a confidence score. No text, nothing to parse.
Look at what my lander actually asks its model. A sentence of telemetry, two choice questions — tilt {left, hold, right} and engine {off, half, full} — and a probability per option. That is Jev's contract. Laya sits behind a single call, agent.predict(prompt, QUESTIONS), and nothing below it knows what produced the answer. Swapping in a decision model would be an adapter, not a redesign.
I have not measured Jev, and its speed, price and “cannot hallucinate” claims are the vendor's, not mine. But my results already answer some of what people are excited about, because those parts belong to the architecture rather than the model:
- A faster, better proposer is not a pilot. I just moved a proposer from 80% to 90% and got one extra solo landing in ninety. Cheaper proposals do not change that.
- Well-typed is not feasible. Every command Laya emits is already well-formed —
left/hold/rightis always in range — and 12–17% of them were still overridden because simulating them forward cost more than the plan. A type system rules out unparseable answers, not physically bad ones. - The usual objection has a runtime answer. People worry that a model returning bare numbers hides its reasoning. My shield never asks the model to explain itself; it rolls the proposal forward under learned physics and compares. Verification beats explanation when you can afford to simulate.
- Calibration is the part I actually want, and it is not free. My shield margin is a cost delta on a scale where a plan costs 0 to about 33, with a 1000-unit crash penalty on top. Confidence is a probability. Nothing converts one into the other, and a well-calibrated probability does not supply the conversion. The real step is calibrating the event the shield decides: the chance that a proposal's best rollout exceeds the plan by more than a given gap.
Two things I would check before assuming a fit. My prompt is almost entirely numbers, and weakness on numbers is a reported limitation of this model class, so interface fit and content fit are separate questions. And a hosted API gives up the offline, on-device property that this project currently has, with roughly 14 ms per decision on a Mac and no network at all.
The usual caveats
The beam search is incomplete and the cost weights are hand-picked. The evaluation is ten held-out seeds per pad. One honest wrinkle: the original cost weights were tuned on seeds 0–9, but the new switching weight was swept on seeds 3000–3005, which sit inside the evaluation set. Every weight in that sweep landed every flight, so the choice turned on jerk and fuel rather than landings, and the landing counts are unchanged from before the penalty existed — but it is tuned on that set, and the results page says so rather than leaving it implicit. None of this is flight software.
Next up is still DAgger, and now calibration has a concrete definition to aim at. The browser lab, the measured results, the training recipe and the full write-up are at github.com/mraad/lunar-mpc-laya. Go fly it and tell me where it breaks.
Comments