Where Is Jev? Find the Tile, Then Detect the Tank

Remember Where's Waldo? You do not look at the whole page at once. You scan it left to right, top to bottom, one cell at a time. When you find the cell with Waldo in it, you look closely and spot him. I wanted to try that idea on an overhead image of an industrial site with a lot of storage tanks.

In my last post, a Jev-style decision model answered a plain-English condition for each row of a table. Here, the rows are image tiles. I cut the image into tiles and ask a vision-language model one fixed question about each tile: does it contain a tank, part of a tank, or no tank? If the answer is no tank, I skip the tile. If not, the tile goes to SAM 3, and SAM 3 finds the tanks. No model is trained for this task.

The project is where-is-jev. In this post, I show how the pieces fit together, what the numbers say, and where the method is wrong.

The data

The test image is one RGB GeoTIFF: 8,200 × 6,087 pixels at 0.5 m per pixel, in EPSG:3857. A FileGDB has 97 tank polygons (the labels) and one extent polygon that covers 82% of the image. 85 labels are inside the extent. I score only the labels and detections that have a centroid inside the extent. The labels are in the repository as data/Tanks.gpkg. The image is not.

How it works

1. Tile. Cut the image into 512 × 512 pixel tiles with a stride of 256 pixels. The tiles overlap by 50%, so each point on the ground is in up to four tiles. This image gives 736 tiles.
2. Gate. Ask the vision-language model the question for each tile. Keep the tiles with answer A or B.
3. Detect. Run SAM 3 with the text prompt “round storage tanks” on the kept tiles. Each SAM 3 mask becomes a polygon. I call this polygon a detection.
4. Merge. Collect the detections from all tiles and remove the duplicates.
5. Evaluate. Match each detection to at most one label, in score order, at IoU 0.5 or more.
6. Package. Write the tiles, detections and scores to one GeoPackage, and open it in ArcGIS Pro.

All model work runs on one Ubuntu machine with four NVIDIA RTX PRO 6000 Blackwell GPUs. Each stage starts one model copy on each GPU, and each copy takes every fourth tile. The merge checks that every tile was processed exactly once before it does anything else.

The gate: one question, one forward pass

The gate asks the same question for every tile:

Classify this RGB overhead satellite image patch for circular industrial storage tanks.
A: Contains at least one complete storage tank whose full circular footprint is visible inside the patch.
B: Contains part of one or more storage tanks cut off by a patch edge or gray missing-image area, but no complete tank.
C: Contains no visible storage tank, either complete or partial.
If both complete and partial tanks appear, choose A. There may be one tank or many.
Partial means spatially cut off, not uncertain. Roads, factory roofs, ships, ponds, and bare ground are not storage tanks.
Gray regions are missing image data. Judge visible evidence only.
Answer with exactly one letter: A, B, or C.

I do not let the model write an answer. For Qwen3-VL, I run one forward pass and read the next-token scores of the letters A, B and C. The letter with the highest score is the decision. For Nemotron, vLLM generates exactly one token from those three letters. There is no classifier head and no fine-tuning. I froze the question before scoring, and I did not tune it on the labels.

I tested four models at three tile sizes (256, 384 and 512 pixels). The reference for a tile comes from the labels: “contains” when a label is fully inside the tile, “partial” when a label only overlaps it. Presence counts both as positive. These are the 512-pixel results on the 513 tiles fully inside the extent:

Gate model	GPUs	Presence precision	Presence recall	Presence F1	Time, 736 tiles
Qwen3-VL 4B	4	83.2%	86.8%	0.849	11.3 s
Qwen3-VL 32B	4	81.0%	91.2%	0.858	31.2 s
Nemotron Nano VL 12B	2	59.2%	92.3%	0.721	45.2 s
Nemotron 3 Nano Omni 30B-A3B	2	69.9%	94.5%	0.804	47.2 s

The times include the model load. Nemotron ran on a second machine with two GPUs and a different runtime, so this is not a fair speed race. The two Qwen models ran on the same machine.

Tiles that the Qwen3-VL 32B gate sends to SAM 3. Green: contains a tank. Amber: partially contains a tank. The tiles with no tank are not drawn.

SAM 3 and the duplicate problem

With the Qwen3-VL 32B gate, 265 of the 736 tiles go to SAM 3. The gate skips the other 64%. SAM 3 (BF16, score threshold 0.3, batch 32) returns 850 detections in about 15 seconds, model load included.

The 50% overlap has a cost. A tank near a tile edge appears in up to four tiles, and some of those copies are cut off by the tile edge. My first merge used box NMS at IoU 0.5. It kept 364 detections, and many of them were duplicates, because the box of a cut-off piece rarely reaches IoU 0.5 with the box of the full tank. Precision was 24.8%.

So I changed the rule. The merge now sorts the detections: first the ones that do not touch their tile edge, then by score. Then it removes a detection when its intersection with a kept detection is more than half of the smaller area. I picked 0.5 before I looked at the scores. This keeps 251 detections. Precision goes up to 36.6%, and recall goes up from 95.3% to 96.5%.

The tile with the most true positives. Left: 45 detections from all overlapping tiles have a centroid in this tile. Right: 11 remain after the merge, and 10 of them match a label.

The results

Gate	Merge	Tiles to SAM 3	Detections	Precision	Recall	F1
Qwen3-VL 4B	New	246	248	36.8%	95.3%	53.1%
Qwen3-VL 32B	New	265	251	36.6%	96.5%	53.1%
Qwen3-VL 32B	Box NMS	265	364	24.8%	95.3%	39.3%

And guess what? The small 4B gate gives the same F1 as the 32B gate. The full run, from the first tile decision to the merged detections, takes about 25 seconds with the 4B gate and about 47 seconds with the 32B gate. In both runs, every one of the 85 labeled tanks touches a detection.

Detections after the merge, Qwen3-VL 32B gate. Green: true positive. Magenta: false positive. Red outline: label without a match.

Where it is wrong

The false positives are not random. 132 of the 142 false positives do not touch any label. In the tile below, SAM 3 finds 40 small round objects in rows. The labels do not include them, and I did not check what they are. If some of them are tanks, the real precision is higher. On the two largest tanks, SAM 3 also splits the roof into parts, and the extra parts count as false positives.

The tile with the most false positives: 40 small round objects in rows that have no label.

Three labels do not reach IoU 0.5. For two of them, the view is oblique, so the image shows the side wall of the tank. The label covers the roof only. The SAM 3 detection covers the roof and the wall. For the third, the detection covers a smaller shape.

The three labels without a match at IoU 0.5. Red: label. Magenta: detection.

Some limits I want to be clear about:

• This is one image and one run for each setup. It does not rank the models in general.
• The gate skips 64% of the tiles. I did not run SAM 3 on all tiles with the same prompt, so I do not have a measured time saving. On four GPUs, SAM 3 is fast on an image of this size.
• The merge can remove a real tank that lies inside a larger false detection. That did not happen to any label here.
• Precision assumes that the labels include every tank inside the extent.

Try it

The launchers have host paths and check the hash of my image, so change them for your machine first. The README lists the model checkpoints and Python environments. Then do these steps on a machine with four GPUs, inside tmux:

1. Clone the repository.

   git clone https://github.com/mraad/where-is-jev.git

2. Go to the project folder.

   cd where-is-jev

3. Run the gate decisions on all GPUs.

   bash run_tiles.sh qwen32b

4. Make the gate folder from the 512-pixel decisions.

   .venv/bin/python patch_decisions.py gate --manifest results/tiles/512/manifest \
     --shards results/tiles/512/qwen32b --out results/gate512

5. Run SAM 3 and the merge on all GPUs.

   bash run_gc1.sh 32 results/sam3_512 results/gate512 "round storage tanks"

6. Evaluate the detections against the labels.

   .venv/bin/python evaluate.py --gdb Tanks.gdb --image Tanks.tif \
     --detections results/sam3_512/merged/detections.geojson --out results/validation

The README has the GeoPackage step, the field of every layer, and the full log of the experiments.

What is next

• Run SAM 3 on all 736 tiles with the same prompt, to measure what the gate saves.
• Find out what the small round objects are.
• Try the gate on more images, and on objects other than tanks.

So far, the Waldo idea holds up: scan the tiles, skip the empty ones, and look closely only where something is there. The code is on GitHub under Apache 2.0. More to come.