Cameras, structure, reprojection error, and the classical pipeline that solves it
Optimisation · feed-forward reconstruction · single-view self-supervision
Poincaré's argument for learning from multiple views
Our new paper on multi-view self-distillation and what emerges from it
"Correspondence, correspondence, correspondence"
Feel free to interrupt with questions.I will also leave room for questions in the end.
To make sure we are familiar with the problem we are trying to solve.
A set of 3D points, surfaces, materials. It exists whether or not anyone photographs it.
Each view has a rigid pose and a calibration. Six photographs are six very different questions asked of the same scene.
A 2D array of intensities. One dimension of the world has been integrated away, and it is the one we want back.
Decades of research. Deep learning has started to transform it.
Find repeatable points in each image and summarise the appearance around them.
Correlate descriptions between a pair of images. Most proposals are wrong.
RANSAC plus a minimal solver. The epipolar constraint discards the outliers and returns R | t.
Intersect the surviving rays. The first 3D points in the reconstruction.
Add the remaining images one at a time, each against the structure so far.
Refine every camera and every point jointly, minimising the gap between each observed x and the predicted π(K[R|t]X).
And nothing is learned, each scene is solved from scratch. The points and cameras contain no higher order features.
Matching is a crucial step. All the subsequent steps depend on matching keypoints.
Reprojection error, per scene, from scratch.
Nothing persists. “Learning” = the reconstruction itself.
No representation beyond cameras and points.
Supervised regression onto 3D pseudo-labels.
A fast approximation of (i), plus latents useful downstream.
A data problem and a weak learning signal.
Self-distillation within one image.
The strongest general-purpose features in vision.
The objective never compares two views.
Note the dependency. (ii) learns from (i)'s output.
SIFT → SuperPoint, DISK, ALIKED. Learned, and better.
SuperGlue, LightGlue, LoFTR, RoMa v2, LoMa. Learned, and much better.
RANSAC + minimal solvers. Still hand-written, still excellent.
Levenberg–Marquardt, and increasingly differentiable variants.
Learning entered this pipeline at stage 2, correspondence. The geometry stayed symbolic, and it still is. Our own entry in that line is the next few slides, because it sets up a theme I keep returning to.
Schonberger & Frahm COLMAP, CVPR 2016 · DeTone et al. SuperPoint · Sarlin et al. SuperGlue · Lindenberger et al. LightGlue · Sun et al. LoFTR · Edstedt et al. RoMa · LoMa, ECCV 2026
The output is a reconstruction, never a model. Run COLMAP twice on two scenes and nothing was learned in between.
David Nordström1*, Johan Edstedt2*, Georg Bökman3, Jonathan Astermark4, Anders Heyden4, Viktor Larsson4, Mårten Wadenbäck2, Michael Felsberg2, Fredrik Kahl1
1 Chalmers University of Technology 2 Linköping University
3 University of Amsterdam 4 Centre for Mathematical Sciences, Lund University
* equal contribution
The reports of the death of the local feature matcher are greatly exaggerated. Nobody had scaled its training data. When you do, the sparse matcher beats the dense ones.
DaD, off the shelf. We do not train a detector at all.
The DeDoDe architecture, trained with a dual-softmax loss over ground-truth matches, which is InfoNCE on correspondences.
LightGlue's design: nine layers, each one self-attention with RoPE and then cross-attention. Trained after, descriptor frozen.
Not just the last. One model covers a family of operating points in speed and accuracy, and the three sizes differ only in width.
A modern take on the local feature matcher. We simply combine best in class, detector, descriptor and matcher and train on more data.
SAME ARCHITECTURE THROUGHOUT · LoMa-B, FROZEN DeDoDe-B, 100K STEPS, SO THE SCALE SITS BELOW THE CHART
Our training recipe, on its own, is worth three tenths of a point. The data is worth 4.9, the longer schedule 4.4, and width another 3.2.
vs. ALIKED + LightGlue
THREE WIDTHS × THREE EXIT LAYERS · HardMatch TEST, mAA@10px · hover to read
No new architecture, and +13.5 mAA at LightGlue's throughput. The matcher runs once per pair, so throughput decides who uses it.
Sort every pair in a benchmark by how well a state-of-the-art matcher does on it. On MegaDepth the curve is a wall: almost nothing is left unsolved, so what the benchmark measures is sub-pixel accuracy and your choice of RANSAC. Whether the model can match has dropped off the axis.
So we annotated 1000 internet pairs by hand and kept the ones a state-of-the-art matcher was unsure about. A smooth difficulty gradient instead of a cliff, and that is the green curve. How it was built is the next slide.
WxBS IS GENUINELY HARD AND HAS 37 PAIRS, TOO FEW TO RESOLVE A MODEL CHANGE
PAIR DIFFICULTY, SORTED · hover to read · click legend to isolate
Benchmark saturation hid a live research problem. You cannot see progress on an instrument that is already pinned.
A hundred Wikimedia Commons categories, ten pairs kept from each, and every correspondence placed by a person. Sorted into nine groups, between 8 and 28 annotated points per pair.
A hundred categories under permissive licences, picked for diversity rather than for difficulty.
Sample 100 pairs per category and read RoMa v2's confidence map. Keep the pairs where its top confidence lands between 0.3 and 0.9, uncertain but still overlapping. Then check each one by hand for whether it is matchable at all.
As many salient correspondences as the pair allows. 90 categories for test, 10 held back for validation, which is the split every ablation in this talk runs on.
SCORED LIKE WxBS · ESTIMATE F FROM THE MATCHER'S OWN MATCHES, THEN TAKE THE EPIPOLAR ERROR OF THE ANNOTATED POINTS
mAA@10px PER GROUP · GROUP SIZE UNDER EACH AXIS · hover an axis · click legend to isolate
Every ring has the same shape. Viewpoint change, aerial pairs and Doppelgängers are where all four still give up.
VIEWPOINT CHANGE210
TEMPORAL GAP144
ILLUMINATION138
NATURE135
AERIAL86
CELESTIAL51
SEASONAL CHANGE48
DOPPELGÄNGER37
HAND DRAWING35HardMatch · THE PAPER'S OWN FIG. 3a · GROUP SIZE AFTER EACH NAME · ALL IMAGES WIKIMEDIA COMMONS, PERMISSIVE LICENCES
The appearance changes and the geometry does not. On a drawing or a century-old photograph there is no photometric relationship left to exploit.
LoMa-G vs. ALIKED+LightGlue · LOCALISATION, NOT MATCHING
Oxford Day-and-Night is the strictest of the three: the protocol makes the matcher build the SfM model out of daytime images and then localise night queries against it. What it measures is the reconstruction.
LoMa is integrated into COLMAP. It runs as the matcher inside the pipeline that most of the field's ground truth comes out of.
Second-hand, so take it as such: I have heard of people at frontier labs running LoMa to generate training data for feed-forward reconstruction models.
Hold on to this for paradigm (ii). A better matcher is a better labeller, and labels are where the next paradigm gets its supervision.
Amortised, meaning that the per-scene optimisation is paid for once, during training, and absorbed into weights. Unlike (i), this one leaves something behind: latents now being used well outside reconstruction.
On paradigm (i)'s output: COLMAP and sensors supply every pointmap it regresses onto.
The first paradigm that leaves a model behind. Which raises the next question: what is it learning from?
OpenVLA-OFT ON LIBERO, OUR SCENE TOKENS CONCATENATED ONTO ITS INPUT, THE ENCODER FROZEN THROUGHOUT
A reconstruction is a result. Features are an input. The policy was not modified and the encoder was never trained for robotics.
David Nordström1, Xinyue Zhang2, Thibaut Loiseau3, Vincent Lepetit3, Fredrik Kahl1
1 Chalmers University of Technology, Sweden
2 Mobile Perception Lab, ShanghaiTech University, China
3 LIGM, École des Ponts, Univ. Gustave Eiffel, CNRS, France
Matchers lean on frozen DINO. Feed-forward reconstruction models have seen orders of magnitude more 3D. So what is in there, and can we matching people just take it?
FROZEN FEATURES, MEGADEPTH-1500 · hover a bar · click legend to isolate
Feed-forward reconstruction does correspondence internally. Nobody trained it to. It is how the architecture satisfies the loss.
VGGT-Ω GLOBAL ATTENTION FOR ONE QUERY · PEAK CIRCLED · RoMa-Ω, FIG. 2
The peak stays on the building through the middle of the network and spreads across the whole image by the last layer. Layers 5, 12, 18 and 24 are the four we pull features from.
VGGT · MEDIAN SAMPSON ERROR INCREASE, ShapeNet PAIRS · hover a bar
Bratulić et al. locate the attention heads that carry correspondence, zero them out, and re-solve for the fundamental matrix. Random early and late layers cost nothing. Four heads in layer 14 costs 70.5 px, and two heads each across layers 12 to 15 costs 76.9 px.
Bratulić, Mittal, Brox & Rupprecht ECCV 2026
The middle layers are load-bearing. We read correspondence out of them, and switching them off takes the two-view geometry with it.
Warp pixels through VGGT-Ω's own depth and cameras, solve for the essential matrix, and you already edge past RoMa v2 on ScanNet-1500, at 34.2 vs 33.6 AUC@5°, with no matching training at all.
And then it falls over. Trained on temporally adjacent video frames, it has never seen a night-time or cross-modality pair: 29.0 mAA on WxBS against RoMa's 72.6.
So we changed exactly one thing in RoMa v2: DINOv3 out, VGGT-Ω in, reading patch features from layers 5, 12, 18 and 24. Nothing else moves.
| WxBS | HardMatch | RUBIK | |
|---|---|---|---|
| VGGT-Ω raw predictions | 29.0 | 20.2 | 84.3 |
| RoMa | 72.6 | 48.1 | 75.4 |
| RoMa v2 DINOv3 backbone | 64.8 | 46.5 | 85.6 |
| RoMa-Ω VGGT-Ω backbone | 72.9 | 50.0 | 88.0 |
mAA@10px ON WxBS AND HardMatch · AUC@20° ON RUBIK
The cost is about 40% slower than RoMa v2, and considerably more VRAM. The backbone is a billion parameters.
Matching is how a 3D representation gets cashed out. It is also a clean probe of whether one is there at all.
Where COLMAP fails, on textureless or repetitive scenes and with few views, there is no label. The failure is inherited, and the training set is bounded by the labeller.
TWO ATTEMPTED ESCAPES · SYNTHETIC DATA AT SCALE (VGGT) · AN SSL OBJECTIVE BOLTED ON (VGGT-Ω)
A depth map is largely predictable from one image. A pose is a handful of scalars. Neither target forces the network to relate two views to each other, which is the part that is actually hard.
SO EVEN WITH PERFECT LABELS, TRAINING FROM SCRATCH IS SLOW AND UNSTABLE
Both problems point the same way. If the signal matters more than the labels, why keep paying for the labels?
THE PRETEXT TARGET · FOR EVERY PIXEL OF A, WHAT HAPPENS TO IT IN B
Every one of those labels can only be produced by relating two views. No monocular shortcut, and a label per pixel rather than six numbers per frame.
SCANNET ROTATION ERROR · ViT-S · MATCHED 120K-STEP BUDGET
Take VGGT-Ω's predictions as labels. Regress them: 6.63°, barely better than scratch. Turn the same predictions into covisibility: 3.31°.
Loiseau et al. From Visibility to Geometry · here at ENPC · ask Thibaut
Note what the better signal is made of. Covisibility is a relation between views, and restructuring around it beats scaling labels.
Two ways of looking at one image. The teacher gets the large global crops whole. The student gets them with half the patches masked out, plus small local crops, so it has to recognise the scene from a fragment.
The loss is agreement. Both heads emit a distribution over 65k prototypes; the student is trained to match the teacher. The teacher is never trained. It is an EMA of the student, so nothing outside the model supervises anything.
Then freeze it. A linear probe on those features beats task-specific supervised training on segmentation, depth, classification and retrieval, all off one backbone. And it now sits upstream of paradigm (ii): VGGT's encoder is DINO.
Caron et al. DINO, ICCV 2021 · Oquab et al. DINOv2 · Siméoni et al. DINOv3 · Zhou et al. iBOT
This is the strongest result in representation learning. And every crop in it came from one photograph.
They probe well for depth and surface normals. Much less well for multiview consistency.
El Banani et al., CVPR 2024
SE(3) structure can be read out of 2D features. Many of you saw Maks Ovsjanikov present this.
Chen et al., 2026
It is remarkable how much 3D leaks into a model that only ever sees one image.
But leaking is not learning. If the probing community keeps finding faint 3D structure in 2D models, the obvious move is to stop probing for it and put it in the objective.
The probing literature keeps measuring the same quantity. How much 3D survives an objective that was never about 3D.
The issue is not that single-view learning is weak in practice. The information is not in the input.
A motionless observer
cannot acquire
the notion of space.
Space is not given by the senses. It is constructed from the correlation between how you move and how what you see changes. A second view is exactly that motion, and it is the one thing a single-image objective is structurally guaranteed never to get.
We did not find this
argument ourselves. Chen et al., SeeSE3, the paper Maks Ovsjanikov presented here, is what put
Poincaré in front of us, and their SE(3) protocol is the one we borrow later.
Our method is named after him for that reason.
Multi-view consistency is a free, unlabelled signal that is actually about 3D. The dominant SSL paradigm cannot use it.
Emergent multi-view geometry through self-distillation. Trained from scratch, on unlabelled image sequences, with no 3D supervision.
Two things move from left to right: how many views the objective sees, and what it is asked to predict. Everything here is self-supervised. The disagreement is about the target.
Poincar3 is the far corner of both axes: many views, and no pixels in the objective at all.
Completion is ill-posed wherever the two views do not overlap. Nothing in view B says what is behind the occluder in view A. A covisibility label is defined everywhere.
Weinzaepfel et al. CroCo, NeurIPS 2022 · Loiseau, Bourmaud & Lepetit Alligat0r, NeurIPS 2025 spotlight · Nordström, Edstedt, Kahl & Bökman MuM, CVPR 2026
Two of these four have ENPC authors. The covisibility slide earlier was Thibaut's follow-up, and Alligat0r is where that line starts.
David Nordström1, Thibaut Loiseau2, Vincent Lepetit2, Michael Felsberg3, Guillaume Bourmaud4, Fredrik Kahl1
1 Chalmers University of Technology, Sweden
2 LIGM, École des Ponts, Univ. Gustave Eiffel, CNRS, France
3 Linköping University, Sweden ·
4 Univ. Bordeaux, CNRS, Bordeaux INP, IMS, France
Self-distillation across multiple views of one scene, from scratch, with no 3D supervision anywhere. Multi-view geometry emerges out of the objective.
And it is trained by regressing 3D labels, and those labels came out of COLMAP or out of a depth sensor. Which is paradigm (i), one step upstream.
We keep the architecture. Alternating frame-wise and global attention is what VGGT and VGGT-Ω do, and it works. The supervision is what we change.
Same architecture, no labels. That is the question, and I had already put it to someone.
Do you think this could be
trained from scratch?
He was not especially convinced. Which I took as a reasonably strong signal that I would have time to find out.
Two experiments before the method. Both are negative results, and both are load-bearing.
The label is free. Eight views of one scene and a 3D point that all of them see. The reconstruction already knows where it lands in every frame, so the track costs nothing.
Two ways of asking. Nearest neighbour in the feature map, or the maximum of the attention map. Both are reported throughout, and neither trains anything.
PCK@25px. Query patches in the first frame, follow them through the other seven, and count the track points landing inside the ring.
El Banani et al. Probing the 3D Awareness of Visual Foundation Models, CVPR 2024 · ScanNet sequences, eight views
Zero-shot multi-view tracking, on a frozen backbone. No head is ever fitted, so the number describes the representation itself.
VGGT-Ω's self-supervised objective is an MSE loss in feature space between student and teacher. On top of a model that is already strong, it refines. Starting from random weights, it has nothing to hold it up.
A regression loss in feature space exerts no pressure towards structure. Nothing stops the representation drifting somewhere smooth, low-rank and useless.
The same budget, the same architecture, the same data. Only the objective changed.
Wang et al. VGGT-Ω, 2026
FROM SCRATCH, MATCHED BUDGET · ScanNet PCK@25px · hover to read
Lesson one. An objective that refines a strong model need not be able to build one.
SCANNET PCK@25, MATCHED COMPUTE / DATA / ARCHITECTURE
Two and a half points for an entire extra dimension of information.
And naive multi-view self-distillation is worse than simply reconstructing RGB. Access to more views and an objective that can use them are different things.
Lesson two. The multi-view signal is sitting right there and the standard objective walks straight past it.
iBOT-style, over masked patches only. The student sees masked views, the teacher never does.
DINO-style, per frame. Without it the representation does not become 3D-aware.
The teacher receives T extra frames of the scene. They never enter the loss. They only add geometric context through attention.
VIT-L ENCODER + 12-LAYER ALTERNATING-ATTENTION DECODER · 650M PARAMS · 400K STEPS · 8×H200, 3 DAYS
ONE TRAINING SEQUENCE · the T extra frames appear on the third build
No RGB reconstruction anywhere. Nothing in the objective asks the model to encode appearance.
TEACHER ASSIGNMENTS · ROWS ARE THE BATCH, COLUMNS ARE PROTOTYPES · press to iterate
The teacher outputs a distribution over 65,536 prototypes. Nothing in the loss says those prototypes have to be used.
Raw softmax piles the mass onto a handful of prototypes. Every view gets the same answer, the student matches it perfectly, and the loss is satisfied by saying nothing.
Normalise the columns and every prototype receives equal total mass. Normalise the rows and every sample is a distribution again. Alternate three times.
DINOv2 ALSO USES SINKHORN–KNOPP PLUS KOLEO TO PREVENT COLLAPSE
Collapse is the default behaviour. Self-distillation works only because something external forces the targets to stay spread out.
SAME COMPUTE, DATA AND ARCHITECTURE THROUGHOUT · 4×H200 FOR ONE DAY PER ROW (I TO VI)
Three ingredients. Full images instead of crops, an image-level objective, and a teacher that sees more of the scene.
Nobody asked it to match. Third time matching turns up in this talk, and the first time nothing supervised it.
RGB reconstruction has to encode appearance alongside geometry, and the responses smear. Self-distillation is free of that constraint, so the representation can spend its capacity on where things are.
Geometry without appearance. The features sharpen as the pixels are taken away.
RE10K MULTI-VIEW RELATIVE POSE, AUC@30°, FULL FINETUNE
It reaches AUC@30 above 65 after only 10k steps at a low learning rate. What was missing from supervised 3D was a representation to start from.
RE10K AUC@30 DURING FINETUNING · hover to read
Paradigms (ii) and (iii) compose. Multi-view SSL is the pretraining that feed-forward reconstruction was missing.
Fit a lightweight Poincaré adapter per scene and ask whether a feature displacement predicts the se(3) twist between two poses. If it does, the feature space has unrolled into something where motion is a straight line.
20 HELD-OUT SCANNET++ SCENES · AVG. R2 ON HELD-OUT PAIRS · PROTOCOL FOLLOWING SEESE3
The last two rows are the same weights. The only difference is whether the model is allowed to see the motion.
Poincaré's claim, as an ablation. Deny the model motion and the notion of space degrades.
No 3D labels means no dependence on paradigm (i). Training can use raw internet video such as SpatialVID and RealEstate10K alongside the annotated sets, and the unlabelled data is where the gains come from.
Poincar3 matches DINOv3 on multi-view correspondence after one day on 8×H200.
MULTI-VIEW CORRESPONDENCE ACCURACY VS. TRAINING COMPUTE
This is what going self-supervised buys. The ceiling stops being the labeller and becomes the compute.
Three directions with a claim on being the convergent objective, and none of them has won yet.
“Correspondence, correspondence, and correspondence.” Takeo Kanade, on the three most important problems in computer vision
The only learned stage. Everything else is solved mathematics, and nothing downstream recovers from a bad correspondence. LoMa · +18.6 mAA
Nobody trained it to. Linear-probe matching beats DINO, and attention knockout destroys epipolar geometry. RoMa-Ω · Bratulić et al.
And it transfers. 94.9 PCK@50 by attention alone, above models trained with 3D labels, and the ones that match best transfer best. Poincar3
Correspondence is the operation that turns images into geometry. All three paradigms keep rediscovering it.
No convergence yet. 3D vision is somewhere around pre-2018, and the objective is still unclaimed.
Much simpler and scales very well.
Language understanding seems to "just work" without any tricks.
Perhaps language can be combined with 3D vision to stabilize the self-supervision.
Trillion parameter LLMs use vision encoders that are a few hundred million parameters.
Vision is far from solved. We are all waiting for a simple pre-training objective. Or, video models are already the answer but I refuse to believe it.
Poincar3 is joint work with collaborators at Chalmers and ENPC.
github.com/davnords/poincar3