OmniPoint: Universal Monocular Metric Pointcloud from Any Camera

Abstract

Recovering metric 3D geometry from monocular images is a fundamental computer vision task, yet current methods remain heavily fragmented by fixed camera model assumptions and inflexible input schemes. We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors, including pinhole, fisheye, and equirectangular projections, while accommodating varying geometric priors. To overcome projection rigidity, OmniPoint abandons conventional planar depth regression. It instead adopts a decoupled ray and distance representation alongside a decoupled training objective, explicitly separating the camera projection model from the scene structure. To address the severe scarcity of training data for alternative cameras, we introduce a bidirectional augmentation strategy that explicitly bridges labeled perspective data and unlabeled omnidirectional domains in 3D space. Furthermore, to seamlessly integrate optional inputs like camera intrinsics or sparse depth without destabilizing the network through feature distribution shifts, we propose a robust information injection mechanism. This mechanism utilizes learnable input state embeddings to resolve architectural ambiguity and applies vectorized Gaussian smoothing to densify irregular measurements. Extensive experiments demonstrate that OmniPoint achieves state-of-the-art zero-shot performance across multiple benchmarks, establishing a robust new standard for unified monocular 3D reconstruction.

Camera-agnostic ray + distance

Both usual prediction targets are tied to the camera. Planar depth z assumes a linear pixel→ray map, diverges as the field of view approaches 180°, and is undefined behind the camera; raw (x, y, z) forces one network to memorise a different projection for every camera type. We predict a unit ray r and a radial distance d instead: the ray absorbs all camera geometry, the distance carries pure scene structure.

$$\mathbf{P} = z \cdot \mathbf{K}^{-1}[u,\, v,\, 1]^{\top}$$

$$\mathbf{P} = d \cdot \mathbf{r}$$

Decoupled training objective

Supervising the two predictions jointly lets a wrong ray penalise a correct distance. We supervise them separately, projecting the predicted distance along the ground-truth ray so that structural error is isolated.

$$\mathcal{L}_{\text{ray}} = \sum_i \big\| \hat{\mathbf{r}}_i - \mathbf{r}_i \big\|_1$$

$$\mathcal{L}_{\text{point}} = \sum_i \big\| s^{*}\hat{d}_i \cdot \mathbf{r}_i - d_i \cdot \mathbf{r}_i \big\|_1$$

Representation, Rel↓	S.FoV	L.FoV	360°
Ray + D (ours)	4.04	6.37	5.79
XYZ point map	4.28	6.74	6.28

Point loss	Point Rel↓
P + GT ray (ours)	5.55
naive P	5.65
disjoint D + Ray	7.88

Both design choices are validated by ablation: the decoupled representation wins on every camera type, and the decoupled loss gives the most accurate point clouds.

Method overview

OmniPoint pipeline: image tokens plus optional intrinsic and sparse-depth tokens and learnable state embeddings enter a DINOv2 ViT-L; DPT heads predict ray and distance, a sky mask, and the global metric scale. — Image tokens + optional intrinsic / sparse-depth tokens → one DINOv2 ViT-L → DPT heads for ray + distance, sky mask and metric scale.

Optional priors, without destabilising the backbone.

Input-state embeddings. Learnable tokens signal which priors are active, resolving the architectural ambiguity that otherwise shifts ViT feature distributions when a prior is toggled on or off.

Vectorised Gaussian smoothing. Each sparse measurement becomes a Gaussian with $\sigma \propto$ depth, scatter-added into a dense, spatially consistent guide plus a binary validity mask — raw irregular points give noisy gradients. $$w(\Delta u, \Delta v) = \exp\!\left(-\frac{\Delta u^{2} + \Delta v^{2}}{2\sigma^{2}}\right)$$

Conditioning design (sparse depth)	Rel↓	δ₁↑
Ours (full)	3.21	98.2
w/o Gaussian smoothing	3.80	97.5
w/o input-state embedding	3.38	97.7

Trained with DINOv2 ViT-L and DPT heads on 29 labelled and 2 unlabelled datasets, on 72 A100s.

Bidirectional data augmentation

Ground-truth 3D for non-pinhole cameras is scarce. Rather than augmenting in 2D, we bridge labelled perspective data and unlabelled omnidirectional data in 3D, in both directions.

Perspective-to-Any. Unproject labelled pinhole data to 3D and re-project it through virtual fisheye and equirectangular cameras; a validity mask removes occlusion holes, giving exact synthetic supervision where none exists.

Any-to-Perspective. Sample one virtual pinhole patch from unlabelled in-the-wild wide-FoV images, pseudo-label it with our pinhole-strong model, and stitch it back to enforce cross-camera consistency.

P2A	A2P	360° Rel↓	360° δ₁↑
–	–	8.45	89.8
✓	–	6.21	94.5
–	✓	8.21	90.9
✓	✓	5.79	95.1

The two directions are complementary: together they cut 360° error from 8.45 to 5.79 Rel.

Qualitative comparisons

Qualitative comparison on panoramic, fisheye and pinhole images against UniK3D, MoGe V2 and UniDepth V2. — Panorama (top), fisheye (middle) and pinhole (bottom). Vertical walls stay vertical and the fisheye road stays straight, where UniK3D's spherical-harmonic ray field bends them.

Panorama comparison against the panorama-specialist DA squared, showing sharper thin structures and undistorted vertical lines. — Against the panorama specialist DA²: thin structures survive and vertical lines stay undistorted.

Quantitative comparisons

One model covering every output, camera and prior

Capability	DA v2	MoGe V2	Depth Pro	UniK3D	DA²	PromptDA	OmniPoint
Affine-inv. depth	✓	✓	✓	✓	–	✓	✓
Affine-inv. points	–	✓	✓	✓	–	–	✓
Metric points	–	✓	✓	✓	–	–	✓
Pinhole	✓	✓	✓	✓	–	✓	✓
Fisheye	–	–	–	✓	–	–	✓
360° panorama	–	–	–	✓	✓	–	✓
Intrinsics prior	–	–	–	✓	–	–	✓
Sparse-depth prior	–	–	–	–	–	✓	✓

A subset of Tab. 1 in the paper.

Zero-shot relative geometry — Rel↓ (%)

Method	S.FoV		L.FoV		360° Images
Method	Depth	Point	Depth	Point	Depth	Point
UniDepth V2	4.23	6.06	7.81	18.4	19.3	102.8
Depth Pro	5.28	7.49	26.6	36.2	–	–
MoGe V1	4.05	5.25	9.48	28.1	22.3	102.6
MoGe V2	3.98	5.59	12.0	23.4	25.1	102.8
DA3	4.78	6.33	12.8	29.9	25.5	101.5
UniK3D	4.14	5.61	8.77	11.5	10.4	11.4
DA²	–	–	–	–	6.65	7.30
OmniPoint (ours)	4.04	5.55	6.37	6.66	5.79	5.90

Small-FoV: 8 pinhole benchmarks · Large-FoV: KITTI360 fisheye · 360°: Stanford2D3D-S + PanoSUNCG. A subset of Tab. 2 in the paper.

Zero-shot metric depth — δ₁↑ (%)

Method	NYUv2	KITTI	ETH3D	iBims-1	DIODE	HAMMER	Mean
UniDepth V2	92.8	95.4	69.5	93.2	51.8	46.8	74.4
Depth Pro	91.9	38.3	32.8	81.5	37.7	63.0	57.5
MoGe V2	96.1	62.9	90.8	83.0	66.4	65.6	77.5
UniK3D	94.4	93.6	83.7	92.8	73.0	58.3	82.6
OmniPoint (ours)	86.2	88.0	91.8	87.7	73.3	73.8	83.5
+ intrinsics	88.3	90.7	93.0	88.3	75.2	74.5	85.0
+ sparse depth	98.5	98.4	94.1	99.0	98.4	99.3	98.0

Bold marks the best of the image-only block; the rows below the rule add optional priors. Providing sparse depth turns the same model into a densification system. Tab. 3 in the paper.

BibTeX

@inproceedings{ye2026omnipoint,
  title     = {OmniPoint: Universal Monocular Metric Pointcloud from Any Camera},
  author    = {Ye, Botao and Pollefeys, Marc and Yang, Ming-Hsuan and Kundu, Abhijit},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

OmniPoint

Universal Monocular Metric Pointcloud from Any Camera