FELN LoRA: Teaching a Small Model to Query the North Sea

In the FELN RAG experiment, I used a few relevant examples to help a language model translate a question into a spatial query. That got me interested in taking the next step: teaching a small model to do this particular job, then running it locally.

I believe small language models have a big role to play in GIS. We have plenty of tasks with a well-defined vocabulary, a known catalog, and a very specific output. Turning a North Sea question into a query over wells, pipelines, and discoveries is one of them. That is the experiment behind feln-lora.

The model is NVIDIA's Nemotron-3-Nano-4B, fine-tuned using LoRA and QLoRA through NeMo AutoModel. Training happens on an NVIDIA GPU. The resulting model runs on my Mac through llama.cpp and Metal, with the question staying on the machine.

Now the best part: the latest local models produce a query in about 1.2 seconds at the warm median. That is a useful place to start for an interactive GIS workflow :-)

Consider this question from the challenge set: “Find gas wells no more than 5 km from oil pipelines.” The expected FELN is:

{
  "layers": ["Wells", "Pipelines"],
  "where": ["\"content_type\" = 2", "\"pipelinestype\" = 4"],
  "relations": ["withinDistance 5 kilometers"]
}

There is quite a bit of GIS knowledge hiding in that little object. Gas wells use content_type = 2, while oil pipelines use pipelinestype = 4. The first layer is what we want returned. The second supplies the spatial filter. Reverse the layers and we are asking for pipelines instead of wells.

layers-json supplies the catalog, and feln supplies the query model and strict comparison. The inference bundle carries the prompt, catalog, and JSON-schema grammar alongside the exported model. The grammar constrains the output structure; getting the right fields, codes, and spatial relationship remains the model's job. Query execution is a separate step, with a read-only DuckDB path in the repository.

LoRA trains small adapters while keeping the base model frozen. QLoRA adds a quantized base during training, using 4-bit NF4 here. Both recipes use rank-16 adapters. After training, I merge the adapter into the full-precision base and export a Q8_0 GGUF for local inference. The 4-bit training representation and the 8-bit deployment file are two separate choices.

For the September 16 retraining, the data preparation produced 3,463 training examples, 410 validation examples, and 419 test examples. A target query and its paraphrases stay in the same split. Otherwise, a reworded training question could make the validation score look more impressive than it deserves. Validation selects the checkpoint; the test and challenge sets are reserved for regression checks.

Here are the v2 results, with each training run using one RTX PRO 6000:

Measurement	LoRA	QLoRA
Merged model, validation	409/410 (99.76%)	409/410 (99.76%)
Q8_0 GGUF, Mac validation	408/410 (99.51%)	408/410 (99.51%)
Q8_0 GGUF, Mac challenge set	37/40	36/40
Warm median query generation, Mac	1.21 seconds	1.16 seconds
Training wall time	28 minutes 22 seconds	31 minutes 6 seconds
Peak GPU memory during training	15.6 GiB	18.4 GiB

The score uses feln.FELN.same, which normalizes SQL formatting while preserving distinctions such as operators, spatial direction, and an empty string versus NULL. A query returning some rows is not enough to make it correct. And these timings measure query generation, with database execution still to follow.

QLoRA gave me the same validation accuracy, but used more memory and took longer in this setup. That is worth measuring on the hardware and model we actually intend to use. The harder, hand-written challenge set also gives a more modest picture than the validation percentage. Prefix matching, category distinctions, and a query combining two spatial relationships still exposed mistakes. I have not measured accuracy against a real-user query log yet.

There is also the question of maintenance. When the catalog aliases, hints, and question phrasing changed, the original models scored 832/862 on the new geometry-layer questions before retraining. The preparation pipeline excludes standalone tables without geometry, including 138 questions from the regenerated gold set. Supporting those tables is still an open extension.

RAG makes it convenient to update examples and catalog context. Fine-tuning adds a training and validation cycle, but gives me a local model specialized for this workflow. The recorded RAG and fine-tuning scores use different evaluation sets, so they do not establish a head-to-head winner.

What excites me is having this translation step close to the data, on the machine in front of me. The code and training notes are available on GitHub in feln-lora. One experiment I would like to try next is adding retrieved examples to the fine-tuned model's prompt. There is room to combine the two approaches. More to come!