Apollo Spatial VLM: From One Photo and One Sentence to 3D

One photo, one sentence. “Locate the Sydney Opera House”, and a few seconds later the scene is in 3D, as Gaussian splats you can tilt around.

Recently I came across a small project called Odyseus Spatial VLM by Mercurius Technologies. Two things grabbed me right away. The first was the name. I grew up on Greek mythology, and seeing Odysseus on a GitHub page brought back a lot of childhood memories. The second was the idea: take a vision language model (VLM), the kind of model that can look at a picture and answer questions about it, and ask it not just what is in the photo, but where it is, in 3D, in meters.

I liked it so much that I forked it, played with it for a while, and kept adding things. It grew into its own project, which I named Apollo Spatial VLM. (Why Apollo? Mercury and Apollo have a lovely story together. I tell it at the end of the README.)

How it works

You upload a photo and type what you are looking for, something like “select the chair near the desk and the closest door”. Then three things happen:

• A VLM looks at the image and points at each thing you asked for. It answers with an x/y position for every target.
• A depth model estimates how far away every pixel is, in meters, from that single photo. No stereo camera, no LiDAR.
• The two are combined: each target's pixel gets its depth, and a simple camera model lifts it into 3D. The browser then shows the whole scene as a point cloud with labeled markers at the targets.

That last step is the whole point. A language model that says “the chair is on the left” is nice. A model that says “the chair is 3.1 meters away, slightly to the left and below the camera” is something a robot or a drone can act on.

What I added

The part I had the most fun with is Gaussian splats. Instead of drawing each depth pixel as a tiny dot, the browser builds a small, flat, soft disc for every pixel, oriented along the surface it sits on. About 92,000 of them for the Opera House photo, built in roughly 20 ms with Spark on top of Three.js. All on the client side, no extra server work. The result looks a lot more like a solid scene than a cloud of dots, as you can see in the animation above.

The other additions are more practical:

• A choice of models in the page. Depth Anything 3 or MoGe-3 for depth; Qwen3.6-27B, Qwen3-VL-32B or Molmo2-8B for picking the targets.
• The camera's own focal length. If the photo carries EXIF data, the demo uses the real field of view instead of guessing 60°. This matters more than I expected (more on that below).
• A lighter response. The server now sends the depth as half floats and lets the browser build the point cloud, which took each response from 2.4 MB down to 0.47 MB.

Testing the models

I had access to a machine with four NVIDIA RTX PRO 6000 Blackwell GPUs, so I put all four to work and ran a small evaluation against public ground truth: 200 referring expressions from RefCOCO for the VLMs, and the 771 laser-scanned DIODE validation images for the depth models.

And guess what? Once you can read their answers, the VLMs all point about equally well, landing inside the right box 88 to 91% of the time. The real difference was whether a model followed the demo's answer format. Qwen3-VL-32B kept wanting to answer with a box instead of a point, and broke the format in about 15% of its answers. Qwen3.6-27B followed it much better, and Molmo2-8B matched the best accuracy at almost three times the speed.

On depth, MoGe-3 was clearly better at getting distances right in meters. But the bigger lesson was about the camera. When I fed Depth Anything 3 a narrower view than the 60° it assumes, its distances collapsed. Given the real focal length, it recovered, and even did a little better than MoGe-3 on that test. That is why the EXIF focal length is now used whenever a photo has it.

Speed-wise, almost all of the time (about 95%) is the VLM thinking. Turning on CUDA graphs in vLLM and switching to FP8 weights brought a full request with Qwen3.6-27B from 3.9 seconds down to about 2 seconds, which is what you see in the animation.

It runs at the edge too

I also executed the whole pipeline on an NVIDIA Jetson AGX Orin (64 GB), the kind of board that sits on a robot or a drone. Everything runs on the device itself: the vision language model, the depth model and the web app. With Molmo2-8B and Depth Anything 3, a full request takes about 5 seconds and uses under half of the Orin's shared memory. MoGe-3 runs there too, at about 9 seconds per request. The bigger Qwen3.6-27B also fits, but at about 16 seconds per request it is better suited to the data center.

A word of honesty

“Metric depth from a single photo” is an estimate, not a measurement. In the Opera House example the depth model puts the building about 5 meters from the camera, which is clearly not true for a photo taken from that far away (unless the Opera House shrank). The model has no idea how big things are in that scene, and it has no focal length to work with since that image carries no EXIF. Indoor shots of rooms and furniture behave much better. Also, 200 samples and one depth dataset are a good sanity check, not a benchmark. And the splats are flat discs built from one view, not a trained 3D Gaussian Splatting model, so walk too far around the scene and you will see the holes behind things.

What is next

The idea I am most excited about: many phone photos also carry a GPS position and a compass heading. Combined with the focal length and the depth, every point in the cloud could get a real latitude and longitude, with an error estimate. That would turn “the chair is 3 meters away” into “the chair is here, on the map”. I wrote up the idea in the repository, and that is where I am heading next.

The source code, the setup scripts for Linux, macOS and Jetson, and the full evaluation are all in the Apollo Spatial VLM repository. A big thank you to Mercurius Technologies for the original Odyseus project. It was the spark that started all of this.

More to come :-)