Skip to content

Preserve Fisheye624 Jacobians on the optical axis - #3791

Draft
Afloat16 wants to merge 1 commit into
nerfstudio-project:mainfrom
Afloat16:fix-fisheye624-optical-axis-gradients
Draft

Afloat16 wants to merge 1 commit into
nerfstudio-project:mainfrom
Afloat16:fix-fisheye624-optical-axis-gradients

Conversation

@Afloat16

@Afloat16 Afloat16 commented Oct 2, 2026 •

Copy link
Copy Markdown

Fisheye624 projection and unprojection return finite values on the optical axis, but their inactive radial branches divide by zero. Those 0/0 intermediates make the coordinate and camera-parameter gradients NaN. The projection's small-radius branch also replaces the ray direction with (1, 1).

Express projection as a scalar radial scale times the normalized coordinates, with the analytic limit theta_distorted / r -> 1. Protect the inactive denominator in the existing unprojection center branch as well. This restores the center Jacobians: diag(fx/z, fy/z) for projection and diag(1/fx, 1/fy) for unprojection.

Validation:

  • Native production imports, including the TorchScript unprojection helper: 13 focused tests passed on CPU Torch 2.14.1; unchanged source: 12 failed, 1 passed. The complete existing tests/cameras directory plus these regressions also passed: 22 passed (existing deprecation warnings).
  • Tests cover both 15/16-parameter layouts, float32/float64 center Jacobians, camera-parameter gradients, gradcheck, near-axis direction, distorted batched round trips with gradients, and a noncentral closed-form control.
  • Repository Ruff 0.12.2 lint/format checks for changed files, license-header check and git diff --check passed.

The complete own-fork workflow was compared with exact unchanged upstream using identical environments. With latest dependencies, both have the same 19 Open3D type errors, two image-processing failures, missing-EGL collection error and missing-pkg_resources documentation error; baseline 79 passed versus patch 92 passed. License, notebook metadata, Ruff lint/format and tracked-file checks pass in both.

A separate compatibility matrix resolves NumPy 1.26.4, setuptools<81 and Open3D 0.18.0 with the dev/docs dependencies, installs libEGL1 and passes uv pip check. Both now pass Pyright (0 errors) and documentation. Baseline 81 passed versus patch 94 passed, including all 13 new regressions; both retain the same two existing training-test failures in pil_to_numpy / Pillow 12.3.0 encoder setimage. Full suites remain red. Compatibility changes are confined to the own-fork validation workflow.

GPU tests and reconstruction-quality benchmarks were not run. Local focused pytest overrides the parallel/type-checking addopts; the comparative workflows use the repository pytest configuration.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant