Tracked pose renders behind the camera (OpenCV vs OpenGL convention); matrixGL_RH throws

[kalwalt]

Code reference: the relevant tracker/pose code is on the dev branch.

Summary

When a marker is successfully tracked, the tracker returns a valid pose (real rotation + translation), but a 3D object placed with it ends up behind the camera and is frustum-clipped, so nothing renders. The right-handed GL matrix (arglCameraViewRHf / matrixGL_RH) that should be used for OpenGL/three.js rendering instead throws an exception.

Discovered while building a static-image Teblid example downstream: marker tracks correctly (Marker tracked ! Num. matches : N) and a found pose arrives, but no overlay is visible.

Downstream references:

• 3D pose renders behind the camera (CV vs GL convention); matrixGL_RH throws webarkit-testing#31 (full analysis)
• (feat): static image example with Teblid tracker webarkit-testing#30 / Add static image example with Teblid tracker (#30) webarkit-testing#32 (the example that surfaced it)

Runtime evidence

Sample pose delivered to the scene (three.js column-major elements):

translation = (0.953, -0.384, +5.824)

Projection matrix from the same run (column-major):

[-1.785, 0, 0, 0,  0, 2.380, 0, 0,  0, 0, -1.0002, -1,  0, 0, -0.20002, 0]

This is a standard OpenGL projection (proj[11] = -1, camera looks down −Z). Transforming the object origin (0.953, -0.384, 5.824, 1):

• clip.w = -view.z = -5.824 → negative (behind the camera)
• ndc.z ≈ 1.03 → beyond the far plane

⇒ the object is clipped and never rasterized, despite correct tracking.

Root cause (hypothesis)

The pose returned to JS is in OpenCV camera convention (object in front ⇒ +Z), which is incompatible with the OpenGL projection used to render (object in front ⇒ −Z). The CV→GL handedness conversion (negate Y and Z on rotation and translation) appears not to be applied to the matrix that reaches the renderer. The dedicated GL matrix path (arglCameraViewRHf → matrixGL_RH) throws, so consumers fall back to the CV-convention pose.

Relevant symbols to look at on dev: cameraPoseFromPoints / pose3d / invertPose / cvToGl, and arglCameraViewRHf.

Suggested direction

• Emit a correct OpenGL modelview/pose matrix (full CV→GL flip), or fix arglCameraViewRHf / matrixGL_RH so it no longer throws and can be fed directly to a GL/three.js scene.
• Quick verification for downstream consumers: negating the translation Z on the returned pose brings the object into view — confirms the convention mismatch.

Related latent bug: first-frame perspectiveTransform crash

WebARKitTracker::processFrame calls GetTrackedFeaturesWarped() when a marker is detected on the very first processed frame (_frameCount == 0), before GetInitialFeatures() has populated the tracked-feature set. perspectiveTransform then runs on an empty point set and throws via CV_Assert(scn + 1 == m.cols) (empty input → default 1-channel Mat → 2 == 3).

This is normally masked with a live camera (detection essentially never succeeds on literal frame 0), but a static image detects immediately on frame 0 and triggers it deterministically. The library should guard the _isDetected || _isTracking pose block against an empty tracked-feature set (or populate the selection on first detection). Downstream worked around it by feeding one blank warmup frame.

Environment

• WebARKitLib dev branch, compiled to WASM (dist/WebARKit.js) in the downstream webarkit-testing repo.
• Tracker type: teblid.