Add spherical_reprojected mode to panorama_sfm example by shuta-ochiai · Pull Request #4552 · colmap/colmap

shuta-ochiai · 2026-07-17T12:48:24Z

Summary

This adds a new mode, spherical_reprojected, to python/examples/panorama_sfm.py, as a middle ground between the existing perspective_* and spherical modes. It implements the hybrid flow proposed in #4406, where @chpatrick agreed this is the right approach:

Render the panoramas into perspective virtual camera images and extract features on them (as in the perspective modes, using the same overlapping layout and masks) — so the feature quality does not suffer from the equirectangular distortion, especially towards the poles.
Reproject the keypoints onto the panoramas (reusing the same spherical mapping as convert_to_equirectangular) and inject them, together with their descriptors and GPS pose priors, into a panorama database with a single native EQUIRECTANGULAR camera.
Match and reconstruct directly on the panoramas (as in the spherical mode), with one image per panorama instead of a rig — so matching and mapping cost stays low (the overlapping rig instead feeds 12 images per panorama into matching and bundle adjustment).

Compared to the flow sketched in #4406, the one difference is that matching runs on the panoramas after reprojecting the keypoints and descriptors, instead of matching on the faces and reprojecting the matches. This is simpler (no match/two-view-geometry rewriting) and lighter: each panorama pair costs a single matcher invocation and a single geometric verification, whereas matching on the faces needs up to N×N face-pair matches and verifications per panorama pair to catch correspondences that cross face boundaries (12×12 for the overlapping layout), with correspondingly more scheduling overhead and database rows. Cross-face correspondences are matched naturally, since all reprojected features of a panorama live in one image.

Implementation notes

The keypoint reprojection math is shared with convert_to_equirectangular via a new PanoProcessor.pano_xy_from_cam_xy helper; rendering + feature extraction is shared with the perspective modes via a new render_and_extract_features helper.
The panorama database skeleton (camera, images, trivial rigs/frames) is created directly via the Database bindings and apply_rig_config([], db), avoiding a throwaway feature extraction pass on the panoramas.
GPS pose priors imported from the renderings' EXIF tags are copied over to the panorama images, so --matcher spatial also works in this mode.
Only the keypoint centers are reprojected; the scale/orientation columns stay in the rendered image frame, which is fine since matching only compares descriptors and mapping only uses the centers.

This mirrors a pipeline we run in production on long 360° video walkthroughs, where the perspective rig mode is accurate but matching/mapping-heavy and the spherical mode loses feature quality; extracting on the renderings and mapping on the panoramas gave us perspective-level feature quality at spherical-level cost.

Closes #4406.

🤖 Generated with Claude Code

shuta-ochiai · 2026-07-17T13:35:49Z

A note on feature/matcher support of the reprojection design (the example itself currently wires up the default SIFT + nearest-neighbor matching only):

SIFT + NN matching (default): supported. The NN matchers only compare descriptors, and mapping only uses the keypoint centers, so reprojecting the centers is sufficient.
ALIKED + LightGlue: should work as-is (untested in this mode). The injection preserves the descriptor dtype/type from the database, and the ALIKED LightGlue model consumes only keypoint positions + descriptors — the reprojected positions are the true panorama coordinates, so its inputs stay consistent. The usual caveat of running LightGlue on equirectangular position distributions applies equally to the spherical mode.
SIFT + LightGlue: not supported yet, future work. The SIFT LightGlue model additionally consumes per-keypoint scale/orientation, which currently stay in the rendered image frame while the positions are in panorama coordinates — i.e. inconsistent inputs. Supporting this would require propagating the local Jacobian of the face→panorama mapping to the shape columns (see the code comment in reproject_features_to_panos).

🤖 Generated with Claude Code

Extract features on the perspective renderings (as in the perspective modes, so the feature quality does not suffer from the equirectangular distortion, especially towards the poles), reproject the keypoints onto the panoramas, and then match and reconstruct directly on the panoramas with the native EQUIRECTANGULAR camera model (as in the spherical mode, with one image per panorama instead of a rig). This gives feature quality comparable to the perspective modes at a matching and mapping cost comparable to the spherical mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sarlinpe · 2026-07-20T06:08:23Z

Thank you for your contribution! I have a few concerns, which to me are for now sufficiently critical to not merge this PR.

Match and reconstruct directly on the panoramas (as in the spherical mode), with one image per panorama instead of a rig — so matching and mapping cost stays low (the overlapping rig instead feeds 12 images per panorama into matching and bundle adjustment).

Could you please provide evidence of this? The speed of matching and BA scales mostly with the number of features, which is unchanged when projecting features from perspective to spherical. BA already considers all observations in the rig at once.

the reprojected positions are the true panorama coordinates, so its inputs stay consistent. The usual caveat of running LightGlue on equirectangular position distributions applies equally to the spherical mode.

Could you please quantify this? I highly doubt that LightGlue performs satisfactorily with such inputs.

…jected-ab # Conflicts: # python/examples/panorama_sfm.py

Resolves the merge conflict between PR colmap#4552 (adds spherical_reprojected mode to the monolithic panorama_sfm.py script) and main's later refactor into pycolmap/panorama.py + thin examples/panorama_sfm.py wrapper. Also wires spherical_reprojected into the TartanAir-v2 benchmark harness (DatasetTartanAirSphericalReprojected) for A/B measurement against perspective_overlapping and spherical.

multiprocessing.Pool's default fork start method breaks the OpenGL/CUDA context that SiftGPU's extractor creates in worker processes, crashing scene evaluation. Force spawn instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

PanoramaReconstructionOptions hardcoded SIFT extraction/matching, so evaluate.py's --feature aliked raised immediately for panorama datasets even though nothing about the reconstruction pipeline actually requires SIFT. Add extractor_type/matcher_type options and wire them through the extraction and matching call sites, and map --feature aliked to ALIKED_N16ROT + ALIKED_LIGHTGLUE in the benchmark harness. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

shuta-ochiai · 2026-08-08T14:48:17Z

Thanks for the review, @sarlinpe — I re-ran a broader benchmark (8 scenes, SIFT+NN and ALIKED+LightGlue, reprojection error, mean track length, GT trajectory accuracy (mAA), and 3DGS novel-view-synthesis quality) to get real numbers on both points instead of relying on intuition. Gaussian count is pinned at 150k across all 3DGS runs — otherwise MCMC densification confounds quality differences with point-count differences.

Quick apology up front: my earlier read was based only on SIFT+NN's 3DGS quality on one dataset, which happened to favor native — too hasty a call on limited data.

Bottom line: ALIKED+LightGlue's 3DGS quality does favor spherical_reprojected fairly consistently, so that part isn't nothing. But I think COLMAP itself should be judged on SfM-native metrics (reprojection error, pose accuracy), not on downstream NVS quality that depends on a specific external renderer/training recipe. On the SfM-native metrics the case is weak: reprojection error is worse for reprojected in all 8 scenes for both feature types, and GT trajectory mAA is at best a wash (near 4/8 split for SIFT+NN, mixed by threshold for ALIKED+LightGlue). Given that, I'm leaning toward withdrawing this PR rather than pushing it forward — details below for the record.

Dataset

TartanAir V2 (CMU AirLab, CC-BY-4.0), specifically the panorama SfM benchmark subset already packaged and released by this repo (benchmark/reconstruction/data/tartanair-v2, GitHub release datasets-tartanair-v2-v1) — not a dataset of my own. All 8 currently-downloaded scenes from the infrastructure category, each 50 panoramas at 2048×1024:

AbandonedFactory-easy-P000, AbandonedCable-easy-P002, AbandonedCable-hard-P003, AbandonedFactory2-easy-P000, AbandonedSchool-easy-P001, CarWelding-easy-P005, CarWelding-hard-P000, CoalMine-hard-P001.

1. Matching cost

Confirmed on 1 scene (AbandonedFactory-easy-P000, the first of the 8 scenes above) that spherical_reprojected matching is ~5.9–6.9x faster than perspective_overlapping for both SIFT+NN and ALIKED+LightGlue, while mapping/BA cost stays roughly unchanged — consistent with your point that BA cost tracks observation count, not image count. This batch (below) only compares spherical vs spherical_reprojected, not perspective_overlapping, so treat the cost claim as directional rather than confirmed at scale.

2. SIFT+NN

Reprojection error / track length, 8 scenes (click to expand)

Scene	Reproj err (sph)	Reproj err (reproj)	Track len (sph)	Track len (reproj)
AbandonedFactory-easy-P000	0.3692	0.4477	7.425	7.009
AbandonedCable-easy-P002	0.3519	0.4026	7.296	6.973
AbandonedCable-hard-P003	0.3865	0.3879	6.375	6.061
AbandonedFactory2-easy-P000	0.3575	0.4163	7.604	7.740
AbandonedSchool-easy-P001	0.3851	0.4130	4.634	4.386
CarWelding-easy-P005	0.5053	0.5225	7.018	7.100
CarWelding-hard-P000	0.4885	0.5307	7.995	7.161
CoalMine-hard-P001	0.4275	0.5195	6.691	6.630
average	0.4089	0.4550	6.880	6.633

Processing time (GPU seconds), 8 scenes (click to expand)

Scene	Extract (sph)	Extract (reproj)	Match (sph)	Match (reproj)	Map (sph)	Map (reproj)	Total (sph)	Total (reproj)
AbandonedFactory-easy-P000	3.76	26.09	2.64	2.24	29.11	25.59	35.51	53.92
AbandonedCable-easy-P002	4.41	28.40	3.45	4.83	34.38	38.74	42.24	71.97
AbandonedCable-hard-P003	4.53	29.41	3.36	7.77	26.60	44.34	34.50	81.52
AbandonedFactory2-easy-P000	4.77	27.60	3.83	3.95	36.85	39.32	45.45	70.88
AbandonedSchool-easy-P001	5.06	28.19	5.96	7.44	16.06	23.42	27.08	59.05
CarWelding-easy-P005	4.90	29.08	3.88	6.40	35.88	38.42	44.65	73.90
CarWelding-hard-P000	4.73	28.98	4.67	8.16	33.02	40.32	42.42	77.46
CoalMine-hard-P001	4.65	27.56	4.79	4.45	29.30	28.68	38.74	60.70
average	4.60	28.16	4.07	5.66	30.15	34.85	38.82	68.68

GT trajectory accuracy (mAA %, IMC-style pose error = max(rotation geodesic error, translation direction error), evaluate.py --error_type relative_auc, gauge-invariant, no GT alignment needed):

GT trajectory mAA %, 8 scenes (click to expand)

Scene	@0.5° (sph)	@0.5° (reproj)	@1.0° (sph)	@1.0° (reproj)	@5.0° (sph)	@5.0° (reproj)	@10.0° (sph)	@10.0° (reproj)
AbandonedFactory-easy-P000	91.27	90.08	95.63	95.04	99.13	99.01	99.56	99.50
AbandonedCable-easy-P002	91.52	90.55	95.76	95.28	99.15	99.06	99.58	99.53
AbandonedCable-hard-P003	91.41	87.27	95.71	91.63	99.14	95.13	99.57	95.56
AbandonedFactory2-easy-P000	93.27	94.16	96.63	97.08	99.33	99.42	99.66	99.71
AbandonedSchool-easy-P001	85.72	83.04	92.86	91.52	98.57	98.30	99.29	99.15
CarWelding-easy-P005	86.07	88.65	91.04	94.32	95.01	98.86	95.50	99.43
CarWelding-hard-P000	83.83	85.08	91.56	92.18	98.22	98.40	99.12	99.20
CoalMine-hard-P001	86.88	91.41	93.42	95.70	98.68	99.14	99.34	99.57
average	88.75	88.76	94.08	94.08	98.40	98.41	98.95	98.96

3DGS quality, 8 scenes (click to expand)

Scene	PSNR (sph)	PSNR (reproj)	SSIM (sph)	SSIM (reproj)	LPIPS (sph)	LPIPS (reproj)
AbandonedFactory-easy-P000	29.465	29.424	0.8874	0.8820	0.1837	0.1929
AbandonedCable-easy-P002	28.807	28.650	0.8997	0.8990	0.1881	0.1916
AbandonedCable-hard-P003	27.336	27.070	0.8613	0.8662	0.2222	0.2159
AbandonedFactory2-easy-P000	27.560	27.764	0.8716	0.8733	0.1687	0.1670
AbandonedSchool-easy-P001	29.132	28.210	0.8282	0.8137	0.3268	0.3427
CarWelding-easy-P005	27.536	27.572	0.8468	0.8508	0.1650	0.1647
CarWelding-hard-P000	28.511	28.761	0.8769	0.8810	0.1350	0.1326
CoalMine-hard-P001	28.643	28.593	0.8508	0.8489	0.2104	0.2133
average	28.374	28.256	0.8653	0.8644	0.2000	0.2026

Native is ahead on reprojection error in all 8 scenes and slightly ahead on average 3DGS PSNR/SSIM, but GT trajectory mAA is a near-exact 4/8 split, and within individual scenes different metrics (PSNR vs SSIM/LPIPS) sometimes disagree on which mode is better (e.g. AbandonedCable-hard-P003: PSNR favors native, SSIM/LPIPS favor reprojected). I'd call this "no meaningful quality regression" rather than "reprojected is clearly as good," which is a weaker claim than I originally suggested.

3. ALIKED+LightGlue

Reprojection error / track length, 8 scenes (click to expand)

Scene	Reproj err (sph)	Reproj err (reproj)	Track len (sph)	Track len (reproj)
AbandonedFactory-easy-P000	0.7829	0.9824	9.973	10.740
AbandonedCable-easy-P002	0.7263	0.8591	10.276	11.722
AbandonedCable-hard-P003	0.5938	0.8074	8.715	9.981
AbandonedFactory2-easy-P000	0.6577	0.9087	10.032	11.300
AbandonedSchool-easy-P001	0.6214	0.7605	4.916	5.466
CarWelding-easy-P005	0.8934	1.1183	9.188	9.901
CarWelding-hard-P000	0.8565	1.0812	11.013	11.581
CoalMine-hard-P001	0.7954	0.9436	8.173	7.935
average	0.7409	0.9327	9.036	9.828

Processing time (GPU seconds), 8 scenes (click to expand)

Scene	Extract (sph)	Extract (reproj)	Match (sph)	Match (reproj)	Map (sph)	Map (reproj)	Total (sph)	Total (reproj)
AbandonedFactory-easy-P000	6.38	27.27	26.49	76.77	21.31	35.70	54.19	139.74
AbandonedCable-easy-P002	7.28	27.62	26.83	88.60	18.42	39.40	52.53	155.61
AbandonedCable-hard-P003	7.11	28.21	27.14	147.42	16.50	46.15	50.75	221.78
AbandonedFactory2-easy-P000	7.72	28.28	26.12	104.08	18.87	44.61	52.71	176.98
AbandonedSchool-easy-P001	7.36	28.63	28.07	87.67	15.17	27.44	50.61	143.73
CarWelding-easy-P005	7.38	28.53	26.75	129.36	18.41	46.38	52.53	204.27
CarWelding-hard-P000	7.34	28.06	27.04	108.65	21.17	42.60	55.56	179.32
CoalMine-hard-P001	6.85	28.35	28.22	70.69	18.22	30.08	53.29	129.12
average	7.18	28.12	27.08	101.66	18.51	39.05	52.77	168.82

Extract includes rendering the 12 perspective views + reprojection for the reproj column. This is spherical vs spherical_reprojected only — native is faster here at every stage, since spherical skips rendering entirely. The relevant cost comparison for your original point (reprojected vs perspective_overlapping) is the N=1 result in section 1 above, not this table.

GT trajectory accuracy (mAA %, see definition above):

GT trajectory mAA %, 8 scenes (click to expand)

Scene	@0.5° (sph)	@0.5° (reproj)	@1.0° (sph)	@1.0° (reproj)	@5.0° (sph)	@5.0° (reproj)	@10.0° (sph)	@10.0° (reproj)
AbandonedFactory-easy-P000	72.09	77.79	86.04	88.90	97.21	97.78	98.60	98.89
AbandonedCable-easy-P002	77.05	82.60	86.53	89.30	94.23	94.76	95.63	95.89
AbandonedCable-hard-P003	65.46	68.78	82.73	82.39	96.55	93.28	98.27	94.68
AbandonedFactory2-easy-P000	55.18	68.35	75.59	84.17	91.92	96.83	94.19	98.42
AbandonedSchool-easy-P001	26.47	37.60	50.44	63.05	86.69	89.41	91.34	92.70
CarWelding-easy-P005	70.17	78.45	84.96	87.18	96.99	94.23	98.50	95.12
CarWelding-hard-P000	76.96	76.70	87.62	85.45	97.39	93.71	98.69	94.84
CoalMine-hard-P001	45.62	50.02	71.17	73.98	94.22	94.80	97.11	97.40
average	61.13	67.54	78.13	81.80	94.40	94.35	96.54	95.99

(50/50 images registered, 1 component, all 8 scenes)

3DGS quality, 8 scenes (click to expand)

Scene	PSNR (sph)	PSNR (reproj)	SSIM (sph)	SSIM (reproj)	LPIPS (sph)	LPIPS (reproj)
AbandonedFactory-easy-P000	28.309	29.230	0.8505	0.8715	0.2560	0.2114
AbandonedCable-easy-P002	28.350	29.013	0.8711	0.8830	0.2301	0.1960
AbandonedCable-hard-P003	26.374	27.525	0.8084	0.8529	0.2926	0.2358
AbandonedFactory2-easy-P000	25.831	26.705	0.8307	0.8460	0.2315	0.1984
AbandonedSchool-easy-P001	27.272	28.922	0.7862	0.8322	0.3885	0.3084
CarWelding-easy-P005	25.709	26.849	0.7937	0.8256	0.2470	0.1848
CarWelding-hard-P000	25.241	27.427	0.8140	0.8535	0.2493	0.1693
CoalMine-hard-P001	26.835	28.059	0.8104	0.8382	0.2754	0.2301
average	26.740	27.966	0.8206	0.8504	0.2713	0.2168

(bold = better; sph = native spherical, reproj = spherical_reprojected)

Reprojected wins clearly at low rotation thresholds (avg @0.5°: +6.4pts, @1.0°: +3.7pts), roughly ties at high thresholds. Mean track length (Reconstruction.compute_mean_track_length()) also favors reprojected in 7/8 scenes, same direction as the 3DGS numbers.

Code change

I also extended PanoramaReconstructionOptions to support choosing the extractor/matcher (it was hardcoded to SIFT, which is why the ALIKED-side numbers above weren't possible before) — pushed as two commits. create_pano_database_from_renderings's existing comment about reprojected keypoint shape being "inconsistent" for learned matchers turned out not to apply to ALIKED in practice (it never populates a non-identity affine shape), updated that comment for accuracy.

Appreciate you taking the time to push on this — it made the evaluation a lot more honest than where I started.

shuta-ochiai force-pushed the panorama-sfm-spherical-reprojected branch 2 times, most recently from 8623562 to eb056c9 Compare July 17, 2026 13:33

shuta-ochiai marked this pull request as ready for review July 17, 2026 13:35

shuta-ochiai force-pushed the panorama-sfm-spherical-reprojected branch from eb056c9 to 60344d5 Compare July 17, 2026 14:26

shuta-ochiai added 2 commits July 18, 2026 00:44

Merge branch 'main' into panorama-sfm-spherical-reprojected

cdbdfe8

Merge branch 'main' into panorama-sfm-spherical-reprojected

2a813dc

Spyder156 mentioned this pull request Aug 2, 2026

What causes the accuracy gap between spherical and perspective panorama SfM? #4541

Open

Merge remote-tracking branch 'origin/main' into bench/spherical-repro…

e60cf23

…jected-ab # Conflicts: # python/examples/panorama_sfm.py

shuta-ochiai force-pushed the panorama-sfm-spherical-reprojected branch from 3b3dfb4 to 9f79e54 Compare August 8, 2026 14:18

shuta-ochiai and others added 3 commits August 8, 2026 23:20

shuta-ochiai force-pushed the panorama-sfm-spherical-reprojected branch from 9f79e54 to 6c08768 Compare August 8, 2026 14:21

Merge branch 'main' into panorama-sfm-spherical-reprojected

f04fa20

shuta-ochiai marked this pull request as draft August 8, 2026 14:49

Navigation Menu

Uh oh!

Add spherical_reprojected mode to panorama_sfm example - #4552

Add spherical_reprojected mode to panorama_sfm example#4552
shuta-ochiai wants to merge 8 commits into
colmap:mainfrom
shuta-ochiai:panorama-sfm-spherical-reprojected

shuta-ochiai commented Jul 17, 2026 •

edited

Loading

Uh oh!

shuta-ochiai commented Jul 17, 2026

Uh oh!

sarlinpe commented Jul 20, 2026

Uh oh!

shuta-ochiai commented Aug 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shuta-ochiai commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation notes

Uh oh!

shuta-ochiai commented Jul 17, 2026

Uh oh!

sarlinpe commented Jul 20, 2026

Uh oh!

shuta-ochiai commented Aug 8, 2026

Dataset

1. Matching cost

2. SIFT+NN

3. ALIKED+LightGlue

Code change

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shuta-ochiai commented Jul 17, 2026 •

edited

Loading