Nano FEL: GPT-2 and nanoGPT on the North Sea

After LoRA on a 4B model and a 1.2B model on the Mac, I kept asking myself a simpler question. How small can the model be, and how plain can the training loop be, before this North Sea task stops working? So I went back to the basics: GPT-2 medium, 355 million parameters, fully fine-tuned with Andrej Karpathy's nanoGPT. No adapters, no chat template, no grammar, no catalog in the prompt. Just a question in, a JSON query out. That is nanofel.

The task is the same FELN translation: a question about wells, pipelines, and discoveries becomes a plan with layers, where, and relations. The twist here is the data format. nanoGPT trains on one flat stream of tokens and samples random windows from it, so each training pair is simply:

Q: Which gas wells in Norway are within 2 kilometers of an oil pipeline?
A: {"layers":["Wells","Pipelines"],
    "where":["content_type = cast(2 as SMALLINT) and (country = 'NO')","PipelinesType = cast(4 as SMALLINT)"],
    "relations":["withinDistance 2 kilometers"]}<|endoftext|>

Pairs are written one after another, and the training script from nanoGPT runs unchanged. The whole project is three vendored files from nanoGPT, a 100-line data preparation script, a normalizer that rewrites generated SQL into the gold surface style, and a 100-line inference script that decodes greedily and stops at the end-of-text token. At inference time the model has no idea a catalog exists; everything it knows about content_type = 2 meaning gas was absorbed during training.

The gold data is 3,000 records from the regenerated NorthSea project, each with a casual phrasing and a canonical phrasing of the same query. I hold out 300 records, which gives 600 evaluation questions, and keep both phrasings of a record on the same side of the split. On top of the gold, about 12,900 generated pairs from the feln-dsl grammar sampler go into training only, after the normalizer puts them in gold style. Without that normalization step the model would learn two competing ways to write the same clause and lose exact match on gold.

Training is short. Block size 256, 32 sequences per iteration, learning rate 3e-5 with cosine decay, dropout 0.1, and 3,000 iterations while keeping the checkpoint with the best validation loss. On two RTX PRO 6000 GPUs that is about five minutes including compile and checkpoint writes. The stripped checkpoint is 1.4 GB and runs on my Mac through MPS.

Here is how the runs went, all scored by exact JSON match on the held-out questions:

Run	Training pairs	Evaluation split	Exact match
v1, gold only	1,800	old gold, 200 questions	158/200 (79.0%)
v1, gpt2-large	1,800	old gold, 200 questions	158/200 (79.0%)
v2, gold + generated	17,372	old gold, 200 questions	194/200 (97.0%)
v3, new gold only	5,400	new gold, 600 questions	529/600 (88.2%)
v4, new gold + generated (shipped)	18,297	new gold, 600 questions	534/600 (89.0%)
v5, same as v4, twice the steps on three GPUs	18,297	new gold, 600 questions	532/600 (88.7%)

Two things jumped out at me. First, going from gpt2-medium to gpt2-large on the same 1,800 pairs changed nothing, while adding generated coverage for the sparse patterns took the same small model from 79% to 97%. This task is data-limited, not capacity-limited, and a 355M model has plenty of room for it. Second, the drop from 97% to 89% is not the model getting worse. The gold set was regenerated with three times more records, and the new catalog hints dropped the layer noun from the subtype descriptions. So “List gas/condensate. The returned wells must be within 15 kilometers of oil.” is now gold for wells near oil pipelines, with nothing in the sentence saying pipelines. 40 of the 66 misses in v4 are that exact situation: the secondary layer is never named, and the model picks discoveries instead of pipelines or the other way around. When every layer is named in the question, v4 scores 95%.

The v5 run was my attempt to squeeze more out of the same data, since the v4 validation loss looked like it was still falling at the last step. Three GPUs, 1.5 times the batch, twice the iterations. The validation loss bottomed out at step 1,800, drifted up while the training loss kept falling, and the best checkpoint landed two questions below v4. Same data, same result. The “still falling” signal was smaller than the evaluation noise, and I should have looked at the eval jitter before spending the compute. The remaining misses are the catalog's ambiguity, and the fix is in the data generation, not in the training loop.

A few caveats. The 89% and the 99.7% from the LoRA post are on different gold sets with different SQL conventions, so there is no head-to-head here. Exact match is strict, and a style-normalized comparison gives the same numbers, so none of the misses are formatting. But the model has no grammar and no output validation. It will happily copy an unfamiliar word into a column name, so a downstream allowlist against the catalog is still needed before anything runs.

What I like about this experiment is how little there is to it. A few hundred lines you can read in one sitting, a five-minute training run, and a model that answers on a laptop. The code, training notes, and the full miss breakdown are on GitHub. Next is regenerating the gold with the layer nouns restored in the hints, and masking the question tokens so the loss is spent on the answer only. More to come!