ENPC, Paris

Learning to
Understand 3D
in the Age of Feed-forward Reconstruction

——  camera centres & frusta
···  sparse 3D structure
     viewing rays
Route

Agenda

01

A primer on Structure-from-Motion

Cameras, structure, reprojection error, and the classical pipeline that solves it

~5 min
02

Three ways to learn 3D

Optimisation · feed-forward reconstruction · single-view self-supervision

~21 min
03

Where current SSL falls short

Poincaré's argument for learning from multiple views

~2 min
04

Poincar3

Our new paper on multi-view self-distillation and what emerges from it

~16 min
05

Future work

"Correspondence, correspondence, correspondence"

~3 min

Feel free to interrupt with questions.I will also leave room for questions in the end.

01
Part one

Background on
3D vision

To make sure we are familiar with the problem we are trying to solve.

Primer · the measurement

Structure-from-Motion

SCENE · 6 VIEWS, SPARSE STRUCTURE
X

The scene

A set of 3D points, surfaces, materials. It exists whether or not anyone photographs it.

R,t

The cameras

Each view has a rigid pose and a calibration. Six photographs are six very different questions asked of the same scene.

x

The image

A 2D array of intensities. One dimension of the world has been integrated away, and it is the one we want back.

Decades of research. Deep learning has started to transform it.

Primer · the classical recipe

How the field has solved it for twenty years

1

Detect & describe

Find repeatable points in each image and summarise the appearance around them.

2

Match

Correlate descriptions between a pair of images. Most proposals are wrong.

3

Verify

RANSAC plus a minimal solver. The epipolar constraint discards the outliers and returns R | t.

4

Triangulate

Intersect the surviving rays. The first 3D points in the reconstruction.

5

Register

Add the remaining images one at a time, each against the structure so far.

6

Bundle adjust

Refine every camera and every point jointly, minimising the gap between each observed x and the predicted π(K[R|t]X).

And nothing is learned, each scene is solved from scratch. The points and cameras contain no higher order features.

Matching is a crucial step. All the subsequent steps depend on matching keypoints.

Part two · the landscape

Three common ways of “learning” 3D

3D lives in the solver
3D lives in the weights
(i)  OPTIMISATION / GEOMETRY

Solve it per scene

COLMAP, GLOMAP, + learned priors
Objective

Reprojection error, per scene, from scratch.

What is learned

Nothing persists. “Learning” = the reconstruction itself.

Limit

No representation beyond cameras and points.

(ii)  FEED-FORWARD RECONSTRUCTION

Amortise the solver

DUSt3R, MASt3R, VGGT, π3
Objective

Supervised regression onto 3D pseudo-labels.

What is learned

A fast approximation of (i), plus latents useful downstream.

Limit

A data problem and a weak learning signal.

(iii)  SINGLE-VIEW SSL

Hope 3D falls out

DINO → DINOv2 → DINOv3, MAE, iBOT
Objective

Self-distillation within one image.

What is learned

The strongest general-purpose features in vision.

Limit

The objective never compares two views.

Note the dependency. (ii) learns from (i)'s output.

Paradigm (i)

Learning-base methods in the classical pipeline

Stage 1Detect & describe

SIFT → SuperPoint, DISK, ALIKED. Learned, and better.

→
Stage 2Match

SuperGlue, LightGlue, LoFTR, RoMa v2, LoMa. Learned, and much better.

→
Stage 3Estimate & verify

RANSAC + minimal solvers. Still hand-written, still excellent.

→
Stage 4Bundle adjust

Levenberg–Marquardt, and increasingly differentiable variants.

Learning entered this pipeline at stage 2, correspondence. The geometry stayed symbolic, and it still is. Our own entry in that line is the next few slides, because it sets up a theme I keep returning to.

Schonberger & Frahm COLMAP, CVPR 2016  ·  DeTone et al. SuperPoint  ·  Sarlin et al. SuperGlue  ·  Lindenberger et al. LightGlue  ·  Sun et al. LoFTR  ·  Edstedt et al. RoMa  ·  LoMa, ECCV 2026

The output is a reconstruction, never a model. Run COLMAP twice on two scenes and nothing was learned in between.

ECCV 2026 · an example from paradigm (i)

LoMa: Local Feature Matching Revisited

David Nordström1*, Johan Edstedt2*, Georg Bökman3, Jonathan Astermark4, Anders Heyden4, Viktor Larsson4, Mårten Wadenbäck2, Michael Felsberg2, Fredrik Kahl1

1 Chalmers University of Technology   2 Linköping University   3 University of Amsterdam   4 Centre for Mathematical Sciences, Lund University
* equal contribution

The reports of the death of the local feature matcher are greatly exaggerated. Nobody had scaled its training data. When you do, the sparse matcher beats the dense ones.

Thread · matching
Paradigm (i) · in practice

Detect, describe, match, with a loss on every layer

1

Detect

DaD, off the shelf. We do not train a detector at all.

2

Describe

The DeDoDe architecture, trained with a dual-softmax loss over ground-truth matches, which is InfoNCE on correspondences.

3

Match

LightGlue's design: nine layers, each one self-attention with RoPE and then cross-attention. Trained after, descriptor frozen.

→

Supervise every layer

Not just the last. One model covers a family of operating points in speed and accuracy, and the three sizes differ only in width.

A modern take on the local feature matcher. We simply combine best in class, detector, descriptor and matcher and train on more data.

Thread · matching
Paradigm (i) · in practice

Nobody had scaled the data

Training mixtureHardMatchmAA@10px
I
MegaDepth, the standard recipe
38.5
II
+ AerialMegaDepth
41.6
III
+ Map-free
43.1
IV
+ BlendedMVS
43.7
V
+ ScanNet++
44.4
VI
+ the remaining mixture, 17 datasets
44.6

SAME ARCHITECTURE THROUGHOUT · LoMa-B, FROZEN DeDoDe-B, 100K STEPS, SO THE SCALE SITS BELOW THE CHART

Our training recipe, on its own, is worth three tenths of a point. The data is worth 4.9, the longer schedule 4.4, and width another 3.2.

+18.6
HardMatch
mAA
+29.5
WxBS
mAA
+24.2
RUBIK
AUC

vs. ALIKED + LightGlue

THREE WIDTHS × THREE EXIT LAYERS · HardMatch TEST, mAA@10px · hover to read

No new architecture, and +13.5 mAA at LightGlue's throughput. The matcher runs once per pair, so throughput decides who uses it.

Thread · matching
Paradigm (i) · in practice

Why anyone thought matching was finished

Sort every pair in a benchmark by how well a state-of-the-art matcher does on it. On MegaDepth the curve is a wall: almost nothing is left unsolved, so what the benchmark measures is sub-pixel accuracy and your choice of RANSAC. Whether the model can match has dropped off the axis.

HardMatch

So we annotated 1000 internet pairs by hand and kept the ones a state-of-the-art matcher was unsure about. A smooth difficulty gradient instead of a cliff, and that is the green curve. How it was built is the next slide.

WxBS IS GENUINELY HARD AND HAS 37 PAIRS, TOO FEW TO RESOLVE A MODEL CHANGE

PAIR DIFFICULTY, SORTED · hover to read · click legend to isolate

Benchmark saturation hid a live research problem. You cannot see progress on an instrument that is already pinned.

Thread · matching
Paradigm (i) · the benchmark

One thousand pairs, annotated by hand

A hundred Wikimedia Commons categories, ten pairs kept from each, and every correspondence placed by a person. Sorted into nine groups, between 8 and 28 annotated points per pair.

1

Scrape

A hundred categories under permissive licences, picked for diversity rather than for difficulty.

2

Let a model find the hard ones

Sample 100 pairs per category and read RoMa v2's confidence map. Keep the pairs where its top confidence lands between 0.3 and 0.9, uncertain but still overlapping. Then check each one by hand for whether it is matchable at all.

3

Annotate

As many salient correspondences as the pair allows. 90 categories for test, 10 held back for validation, which is the split every ablation in this talk runs on.

SCORED LIKE WxBS · ESTIMATE F FROM THE MATCHER'S OWN MATCHES, THEN TAKE THE EPIPOLAR ERROR OF THE ANNOTATED POINTS

mAA@10px PER GROUP · GROUP SIZE UNDER EACH AXIS · hover an axis · click legend to isolate

Every ring has the same shape. Viewpoint change, aerial pairs and Doppelgängers are where all four still give up.

Thread · matching
Paradigm (i) · the benchmark

One pair from each group

Viewpoint ChangeVIEWPOINT CHANGE210
Temporal GapTEMPORAL GAP144
IlluminationILLUMINATION138
NatureNATURE135
AerialAERIAL86
CelestialCELESTIAL51
Seasonal ChangeSEASONAL CHANGE48
DoppelgangerDOPPELGÄNGER37
Hand DrawingHAND DRAWING35

HardMatch · THE PAPER'S OWN FIG. 3a · GROUP SIZE AFTER EACH NAME · ALL IMAGES WIKIMEDIA COMMONS, PERMISSIVE LICENCES

The appearance changes and the geometry does not. On a drawing or a century-old photograph there is no photometric relationship left to exploit.

Thread · matching
Paradigm (i) · in practice

The same matcher, measured downstream

+21.7
Map-free
precision
+37.4
InLoc DUC2
0.25m, 2°
+16.1
Oxford, night
0.25m, 2°

LoMa-G vs. ALIKED+LightGlue · LOCALISATION, NOT MATCHING

Oxford Day-and-Night is the strictest of the three: the protocol makes the matcher build the SfM model out of daytime images and then localise night queries against it. What it measures is the reconstruction.

Where it runs

LoMa is integrated into COLMAP. It runs as the matcher inside the pipeline that most of the field's ground truth comes out of.

And what it is being used for

Second-hand, so take it as such: I have heard of people at frontier labs running LoMa to generate training data for feed-forward reconstruction models.

Hold on to this for paradigm (ii). A better matcher is a better labeller, and labels are where the next paradigm gets its supervision.

Paradigm (ii)

Feed-forward reconstruction

Amortised, meaning that the per-scene optimisation is paid for once, during training, and absorbed into weights. Unlike (i), this one leaves something behind: latents now being used well outside reconstruction.

But trained on what?

On paradigm (i)'s output: COLMAP and sensors supply every pointmap it regresses onto.

The first paradigm that leaves a model behind. Which raises the next question: what is it learning from?

Paradigm (ii) · what it leaves behind

Usefulness of learning to reconstruct

OpenVLA-OFT ON LIBERO, OUR SCENE TOKENS CONCATENATED ONTO ITS INPUT, THE ENCODER FROZEN THROUGHOUT

A reconstruction is a result. Features are an input. The policy was not modified and the encoder was never trained for robotics.

ECCV Workshops 2026 · an example from paradigm (ii)

RoMa-Ω: What Feed-Forward 3D Models Know About Image Matching

David Nordström1, Xinyue Zhang2, Thibaut Loiseau3, Vincent Lepetit3, Fredrik Kahl1

1 Chalmers University of Technology, Sweden
2 Mobile Perception Lab, ShanghaiTech University, China
3 LIGM, École des Ponts, Univ. Gustave Eiffel, CNRS, France

Matchers lean on frozen DINO. Feed-forward reconstruction models have seen orders of magnitude more 3D. So what is in there, and can we matching people just take it?

Thread · matching
Paradigm (ii) · in practice

The matching is in there. It is just hidden.

FROZEN FEATURES, MEGADEPTH-1500 · hover a bar · click legend to isolate

The matching information is in there

Feed-forward reconstruction does correspondence internally. Nobody trained it to. It is how the architecture satisfies the loss.

Thread · matching
Paradigm (ii) · converging evidence

The same layers, found a different way

QUERYL = 5L = 12L = 18L = 24
Query patch marked with a star on the facade Global attention for the query at layer 5, peak on the facade Global attention at layer 12, peak on the facade Global attention at layer 18, peak on the facade Global attention at layer 24, response spread over the whole image

VGGT-Ω GLOBAL ATTENTION FOR ONE QUERY · PEAK CIRCLED · RoMa-Ω, FIG. 2

What the attention map looks like

The peak stays on the building through the middle of the network and spreads across the whole image by the last layer. Layers 5, 12, 18 and 24 are the four we pull features from.

VGGT · MEDIAN SAMPSON ERROR INCREASE, ShapeNet PAIRS · hover a bar

Converging evidence

Bratulić et al. locate the attention heads that carry correspondence, zero them out, and re-solve for the fundamental matrix. Random early and late layers cost nothing. Four heads in layer 14 costs 70.5 px, and two heads each across layers 12 to 15 costs 76.9 px.

Bratulić, Mittal, Brox & Rupprecht  ECCV 2026

The middle layers are load-bearing. We read correspondence out of them, and switching them off takes the two-view geometry with it.

Thread · matching
Paradigm (ii) · in practice

Taking the features instead of the predictions

Training-free, from the raw predictions

Warp pixels through VGGT-Ω's own depth and cameras, solve for the essential matrix, and you already edge past RoMa v2 on ScanNet-1500, at 34.2 vs 33.6 AUC@5°, with no matching training at all.

And then it falls over. Trained on temporally adjacent video frames, it has never seen a night-time or cross-modality pair: 29.0 mAA on WxBS against RoMa's 72.6.

So we changed exactly one thing in RoMa v2: DINOv3 out, VGGT-Ω in, reading patch features from layers 5, 12, 18 and 24. Nothing else moves.

 WxBSHardMatchRUBIK
VGGT-Ω raw predictions29.020.284.3
RoMa72.648.175.4
RoMa v2 DINOv3 backbone64.846.585.6
RoMa-Ω VGGT-Ω backbone72.950.088.0

mAA@10px ON WxBS AND HardMatch · AUC@20° ON RUBIK

The cost is about 40% slower than RoMa v2, and considerably more VRAM. The backbone is a billion parameters.

Matching is how a 3D representation gets cashed out. It is also a clean probe of whether one is there at all.

Paradigm (ii)

Two problems with feed-forward reconstruction

Problem 1 · the data

Where COLMAP fails, on textureless or repetitive scenes and with few views, there is no label. The failure is inherited, and the training set is bounded by the labeller.

TWO ATTEMPTED ESCAPES · SYNTHETIC DATA AT SCALE (VGGT) · AN SSL OBJECTIVE BOLTED ON (VGGT-Ω)

Problem 2 · the learning signal

A depth map is largely predictable from one image. A pose is a handful of scalars. Neither target forces the network to relate two views to each other, which is the part that is actually hard.

SO EVEN WITH PERFECT LABELS, TRAINING FROM SCRATCH IS SLOW AND UNSTABLE

Both problems point the same way. If the signal matters more than the labels, why keep paying for the labels?

Paradigm (ii) · a way out

Covisibility pretraining, a stronger signal

THE PRETEXT TARGET · FOR EVERY PIXEL OF A, WHAT HAPPENS TO IT IN B

Every one of those labels can only be produced by relating two views. No monocular shortcut, and a label per pixel rather than six numbers per frame.

From scratch all 120k steps on depth & cameras
7.70
Covisibility first, then fine-tune
3.13
+ masked modelling
2.73

SCANNET ROTATION ERROR · ViT-S · MATCHED 120K-STEP BUDGET

Controlled ablation

Take VGGT-Ω's predictions as labels. Regress them: 6.63°, barely better than scratch. Turn the same predictions into covisibility: 3.31°.

Loiseau et al.  From Visibility to Geometry  ·  here at ENPC · ask Thibaut

Note what the better signal is made of. Covisibility is a relation between views, and restructuring around it beats scaling labels.

Paradigm (iii)

Self-distillation, in one picture

Two ways of looking at one image. The teacher gets the large global crops whole. The student gets them with half the patches masked out, plus small local crops, so it has to recognise the scene from a fragment.

The loss is agreement. Both heads emit a distribution over 65k prototypes; the student is trained to match the teacher. The teacher is never trained. It is an EMA of the student, so nothing outside the model supervises anything.

Then freeze it. A linear probe on those features beats task-specific supervised training on segmentation, depth, classification and retrieval, all off one backbone. And it now sits upstream of paradigm (ii): VGGT's encoder is DINO.

Caron et al. DINO, ICCV 2021  ·  Oquab et al. DINOv2  ·  Siméoni et al. DINOv3  ·  Zhou et al. iBOT

This is the strongest result in representation learning. And every crop in it came from one photograph.

Paradigm (iii)

So how much 3D is in there already?

Probing the 3D awareness of foundation models

They probe well for depth and surface normals. Much less well for multiview consistency.

El Banani et al., CVPR 2024

SeeSE3 · emergence of 3D space in vision features

SE(3) structure can be read out of 2D features. Many of you saw Maks Ovsjanikov present this.

Chen et al., 2026

It is remarkable how much 3D leaks into a model that only ever sees one image.

But leaking is not learning. If the probing community keeps finding faint 3D structure in 2D models, the obvious move is to stop probing for it and put it in the objective.

The probing literature keeps measuring the same quantity. How much 3D survives an objective that was never about 3D.

02
Part three · the argument

The problem
with one view

The issue is not that single-view learning is weak in practice. The information is not in the input.

The argument

The argument is a hundred and twenty years old

A motionless observer
cannot acquire
the notion of space.

HENRI POINCARÉ  La Valeur de la Science, 1905  ·  paraphrased

Space is not given by the senses. It is constructed from the correlation between how you move and how what you see changes. A second view is exactly that motion, and it is the one thing a single-image objective is structurally guaranteed never to get.

We did not find this argument ourselves. Chen et al., SeeSE3, the paper Maks Ovsjanikov presented here, is what put Poincaré in front of us, and their SE(3) protocol is the one we borrow later.
Our method is named after him for that reason.

Multi-view consistency is a free, unlabelled signal that is actually about 3D. The dominant SSL paradigm cannot use it.

03
Part four · our work

Poincar3

Emergent multi-view geometry through self-distillation. Trained from scratch, on unlabelled image sequences, with no 3D supervision.

Part four · what came before

The lineage Poincar3 comes out of

Two things move from left to right: how many views the objective sees, and what it is asked to predict. Everything here is self-supervised. The disagreement is about the target.

Poincar3 is the far corner of both axes: many views, and no pixels in the objective at all.

Why the target changed

Completion is ill-posed wherever the two views do not overlap. Nothing in view B says what is behind the occluder in view A. A covisibility label is defined everywhere.

Weinzaepfel et al. CroCo, NeurIPS 2022  ·  Loiseau, Bourmaud & Lepetit Alligat0r, NeurIPS 2025 spotlight  ·  Nordström, Edstedt, Kahl & Bökman MuM, CVPR 2026

Two of these four have ENPC authors. The covisibility slide earlier was Thibaut's follow-up, and Alligat0r is where that line starts.

Preprint · the main course

Poincar3: Emergent Multi-View Geometry Through Self-Distillation

David Nordström1, Thibaut Loiseau2, Vincent Lepetit2, Michael Felsberg3, Guillaume Bourmaud4, Fredrik Kahl1

1 Chalmers University of Technology, Sweden
2 LIGM, École des Ponts, Univ. Gustave Eiffel, CNRS, France
3 Linköping University, Sweden  ·  4 Univ. Bordeaux, CNRS, Bordeaux INP, IMS, France

Self-distillation across multiple views of one scene, from scratch, with no 3D supervision anywhere. Multi-view geometry emerges out of the objective.

Poincar3 · setup

The thing we are trying to train

And it is trained by regressing 3D labels, and those labels came out of COLMAP or out of a depth sensor. Which is paradigm (i), one step upstream.

We keep the architecture. Alternating frame-wise and global attention is what VGGT and VGGT-Ω do, and it works. The supervision is what we change.

Same architecture, no labels. That is the question, and I had already put it to someone.

Poincar3 · origin
CVPR · a poster hall · earlier this year

Do you think this could be
trained from scratch?

what I asked Andrea Vedaldi, having just read VGGT-Ω

He was not especially convinced. Which I took as a reasonably strong signal that I would have time to find out.

What “from scratch” has to mean
no 3D supervision, at any stage
no pretrained 3D backbone
no correspondence labels
only unlabelled image sequences of one scene

Two experiments before the method. Both are negative results, and both are load-bearing.

Poincar3 · how it is measured

What counts as understanding 3D

The label is free. Eight views of one scene and a 3D point that all of them see. The reconstruction already knows where it lands in every frame, so the track costs nothing.

Two ways of asking. Nearest neighbour in the feature map, or the maximum of the attention map. Both are reported throughout, and neither trains anything.

PCK@25px. Query patches in the first frame, follow them through the other seven, and count the track points landing inside the ring.

El Banani et al. Probing the 3D Awareness of Visual Foundation Models, CVPR 2024  ·  ScanNet sequences, eight views

Zero-shot multi-view tracking, on a frozen backbone. No head is ever fitted, so the number describes the representation itself.

Poincar3 · what fails

First, the objective that does not survive from scratch

VGGT-Ω's self-supervised objective is an MSE loss in feature space between student and teacher. On top of a model that is already strong, it refines. Starting from random weights, it has nothing to hold it up.

A regression loss in feature space exerts no pressure towards structure. Nothing stops the representation drifting somewhere smooth, low-rank and useless.

The same budget, the same architecture, the same data. Only the objective changed.

Wang et al. VGGT-Ω, 2026

FROM SCRATCH, MATCHED BUDGET · ScanNet PCK@25px · hover to read

Lesson one. An objective that refines a strong model need not be able to build one.

Poincar3 · what fails

Second, just showing DINO more views

Single-view DINOv2 objective baseline
47.2
The same objective, now with multiple views III
49.7
Plain multi-view RGB reconstruction MuM
54.5

SCANNET PCK@25, MATCHED COMPUTE / DATA / ARCHITECTURE

Two and a half points for an entire extra dimension of information.

And naive multi-view self-distillation is worse than simply reconstructing RGB. Access to more views and an objective that can use them are different things.

Lesson two. The multi-view signal is sitting right there and the standard objective walks straight past it.

Poincar3 · method

Multi-view self-distillation

L = Lpatch + α·Lglobal + β·LKoLeo
1

Masked patch distillation

iBOT-style, over masked patches only. The student sees masked views, the teacher never does.

2

Image-level distillation

DINO-style, per frame. Without it the representation does not become 3D-aware.

3

A better-informed teacher

The teacher receives T extra frames of the scene. They never enter the loss. They only add geometric context through attention.

VIT-L ENCODER + 12-LAYER ALTERNATING-ATTENTION DECODER · 650M PARAMS · 400K STEPS · 8×H200, 3 DAYS

ONE TRAINING SEQUENCE · the T extra frames appear on the third build

No RGB reconstruction anywhere. Nothing in the objective asks the model to encode appearance.

Poincar3 · method

What stops it collapsing

TEACHER ASSIGNMENTS · ROWS ARE THE BATCH, COLUMNS ARE PROTOTYPES · press to iterate

The teacher outputs a distribution over 65,536 prototypes. Nothing in the loss says those prototypes have to be used.

The failure mode

Raw softmax piles the mass onto a handful of prototypes. Every view gets the same answer, the student matches it perfectly, and the loss is satisfied by saying nothing.

Sinkhorn–Knopp, three iterations

Normalise the columns and every prototype receives equal total mass. Normalise the rows and every sample is a distribution again. Alternate three times.

DINOv2 ALSO USES SINKHORN–KNOPP PLUS KOLEO TO PREVENT COLLAPSE

Collapse is the default behaviour. Self-distillation works only because something external forces the targets to stay spread out.

Poincar3 · ablation

What the objective is worth

ObjectiveMulti-view matching, ScanNetPCK@25
I
RGB reconstruction, the MuM objective
54.5
II
Single-view DINOv2, the baseline
47.2
III
Multi-view DINOv2, just add views
49.7
IV
Multi-view iBOT, drop the local crops
55.7
V
+ image-level objective
66.7
VI
+ teacher sees more views
70.2
VII
+ scale: Poincar3
83.7

SAME COMPUTE, DATA AND ARCHITECTURE THROUGHOUT · 4×H200 FOR ONE DAY PER ROW (I TO VI)

Three ingredients. Full images instead of crops, an image-level objective, and a teacher that sees more of the scene.

Thread · matching
Poincar3 · results

Correspondence emerges in the attention map

Tracks across frames formed by following the highest-activation attention patch
ZERO-SHOT TRACKS · HIGHEST-ACTIVATION ATTENTION PATCH, NO CORRESPONDENCE OR 3D SUPERVISION
94.9
Poincar3, attention matching
ScanNet PCK@50
83.8
π3, trained with 3D labels
ScanNet PCK@50
72.2
DINOv3, feature matching
ScanNet PCK@50

Nobody asked it to match. Third time matching turns up in this talk, and the first time nothing supervised it.

Poincar3 · results

Dropping pixels sharpens the features

Feature correlation across frames for a query patch: diffuse for RGB reconstruction, localised for ours
QUERY PATCH (STAR) · FEATURE CORRELATION ACROSS FRAMES, MAXIMUM RESPONSE CIRCLED

RGB reconstruction has to encode appearance alongside geometry, and the responses smear. Self-distillation is free of that constraint, so the representation can spend its capacity on where things are.

Geometry without appearance. The features sharpen as the pixels are taken away.

Poincar3 · results

And it is what feed-forward reconstruction wanted all along

Random initialisation from scratch
17.4
DINOv3 initialisation single-view SSL
31.2
Poincar3 initialisation multi-view SSL
68.2

RE10K MULTI-VIEW RELATIVE POSE, AUC@30°, FULL FINETUNE

It reaches AUC@30 above 65 after only 10k steps at a low learning rate. What was missing from supervised 3D was a representation to start from.

RE10K AUC@30 DURING FINETUNING · hover to read

Paradigms (ii) and (iii) compose. Multi-view SSL is the pretraining that feed-forward reconstruction was missing.

Poincar3 · results

Camera motion is linear in the features

ΔPt,s ≈ W ( φ(Ht+s) − φ(Ht) )

Fit a lightweight Poincaré adapter per scene and ask whether a feature displacement predicts the se(3) twist between two poses. If it does, the feature space has unrolled into something where motion is a straight line.

20 HELD-OUT SCANNET++ SCENES · AVG. R2 ON HELD-OUT PAIRS · PROTOCOL FOLLOWING SEESE3

DINOv3 single view
0.046
MuM multi-view, RGB recon.
0.081
Poincar3 multi-view context OFF
0.053
Poincar3 multi-view context ON
0.098

The last two rows are the same weights. The only difference is whether the model is allowed to see the motion.

Poincaré's claim, as an ablation. Deny the model motion and the notion of space degrades.

Poincar3 · results

The data problem dissolves

No 3D labels means no dependence on paradigm (i). Training can use raw internet video such as SpatialVID and RealEstate10K alongside the annotated sets, and the unlabelled data is where the gains come from.

3D-labelled data only ScanNet PCK@50
87.5
+ unlabelled internet sequences
94.9

Poincar3 matches DINOv3 on multi-view correspondence after one day on 8×H200.

MULTI-VIEW CORRESPONDENCE ACCURACY VS. TRAINING COMPUTE

This is what going self-supervised buys. The ceiling stops being the labeller and becomes the compute.

04
Part five · outlook

Future work

Three directions with a claim on being the convergent objective, and none of them has won yet.

Thread · matching
Outlook · the thread

One thing kept showing up in all three

“Correspondence, correspondence, and correspondence.” Takeo Kanade, on the three most important problems in computer vision

The only learned stage. Everything else is solved mathematics, and nothing downstream recovers from a bad correspondence. LoMa · +18.6 mAA

Nobody trained it to. Linear-probe matching beats DINO, and attention knockout destroys epipolar geometry. RoMa-Ω · Bratulić et al.

And it transfers. 94.9 PCK@50 by attention alone, above models trained with 3D labels, and the ones that match best transfer best. Poincar3

Correspondence is the operation that turns images into geometry. All three paradigms keep rediscovering it.

Outlook

Three candidates for the convergent objective

Candidate A

Video generation

  • The closest thing vision has to next-token prediction.
  • Sees motion parallax, so multi-view is in the input for free.
  • Geometry stays implicit and non-metric.
  • Enormous compute for a signal mostly about appearance.
Candidate B

Feed-forward reconstruction

  • Outputs exactly the quantity we want, in one pass.
  • Already the practical default for sparse-view 3D.
  • Bounded by the labeller, and the signal is weak.
  • Better as a head than as a pretraining objective.
Candidate C

Multi-view SSL

  • The only loss here that is both label-free and geometric.
  • Inherits the DINO lineage and adds the missing constraint.
  • Composes with B as the initialisation that makes it work.
  • No DINOv2-scale demonstration yet. My bet is here.

No convergence yet. 3D vision is somewhere around pre-2018, and the objective is still unclaimed.

Outlook

What I do not know

Q1

Should we just be using video models?

Much simpler and scales very well.

Q2

Why do we need all these tricks and engineering?

Language understanding seems to "just work" without any tricks.

Q3

Where does language fit in?

Perhaps language can be combined with 3D vision to stabilize the self-supervision.

Q4

Why are vision encoders so small in LLMs?

Trillion parameter LLMs use vision encoders that are a few hundred million parameters.

Vision is far from solved. We are all waiting for a simple pre-training objective. Or, video models are already the answer but I refuse to believe it.

Thank you

Questions,
please

Poincar3 is joint work with collaborators at Chalmers and ENPC.
github.com/davnords/poincar3

00:00·1 / 33·? for keys
Speaker notes  

Presenting

→  space
Next step, then next slide
←
Back
↓  ↑
Skip whole slides
N
Speaker notes
O
Slide overview
T
Start / reset the 45-min clock
L
Light mode (for bad projectors)
F
Fullscreen
B
Black the screen
?
This card