This guide walks through the complete eye-in-hand calibration and localization pipeline: calibrate your camera's intrinsics, calibrate the hand-eye transform, then locate AprilTag objects in your robot's workspace at runtime.
Before starting you need:
eyehandinstalled (pip install -e .)- A printed ChArUco board with known dimensions (squares and markers measured in metres) — generate a print-ready PDF with
python -m eyehand.scripts.generate_board -o board.pdf(see Step 1 for options) - Printed AprilTags (tag36h11 family) with a known tag size (side length in metres)
- A camera mounted on your robot's end-effector, accessible via a class implementing
CameraInterface - A class implementing
RobotInterfacethat providesget_ee_pose()returningT_ee_in_base— the end-effector pose expressed in the robot base frame (useMockRobotfor testing)
Intrinsic calibration determines your camera's focal lengths, principal point, and distortion coefficients. You need a printed ChArUco board with known geometry.
If you don't already have a board printed, generate one with the bundled script:
python -m eyehand.scripts.generate_board -o board.pdf
# Or customise:
python -m eyehand.scripts.generate_board \
--squares-x 5 --squares-y 7 \
--square-length 0.036 --marker-length 0.027 \
--paper letter \
--dpi 300 \
-o board.pdfDefaults: 5×7 squares, 36 mm target square / 27 mm marker, 300 DPI, DICT_6X6_250, US Letter page, and a printer-scale compensation factor of 0.9444. The board is centered on the chosen page with white margins so the printer always sees a real Letter (or A4) page, eliminating "fit to printable area" surprises. Use --paper a4 for A4, or --paper none to crop the PDF tightly to the board with no surrounding margins.
Why the printer-scale compensation? Many printers — notably the HP LaserJet M110/M111/M112 series and some HP Smart drivers on Windows — silently shrink printed pages by roughly 5.6% even when the print dialog reports "100% / actual size". The --printer-scale flag (default 0.9444) oversizes the generated board so the printed output matches your requested --square-length. If your printer is dimensionally faithful, disable compensation:
python -m eyehand.scripts.generate_board --printer-scale 1.0 -o board.pdfIf your printer has a different shrink factor, measure a known-size print and pass the measured ratio:
python -m eyehand.scripts.generate_board --printer-scale 0.95 -o board.pdf # exampleWhy PDF? The PDF embeds its page size, so printing at "actual size" uses unambiguous physical dimensions. Print at 100 % scale ("actual size" in your print dialog, not "fit to page") and verify a square with calipers — whatever value your calipers read is the authoritative number to pass to CharucoDetector(square_length=…) below, regardless of what the script claimed. Pass -o board.png if you prefer PNG output.
import cv2
import numpy as np
from pathlib import Path
from eyehand import CharucoDetector, IntrinsicCalibrator, CalibrationStore
# Create a detector matching your physical board dimensions
# squares_x / squares_y: number of chessboard squares on each axis
# square_length: side of one square in metres
# marker_length: side of one ArUco marker in metres
detector = CharucoDetector(
squares_x=5,
squares_y=7,
square_length=0.036, # 36 mm squares
marker_length=0.027, # 27 mm markers
)
# Capture 15-20 grayscale images of the board from varied angles:
# - Different distances (0.3 m to 1.0 m)
# - Tilted left/right, up/down (>= 30 deg per axis for good coverage)
# - Board should fill at least 50% of the frame in some images
gray_images = []
for _ in range(20):
frame = camera.capture()
gray = cv2.cvtColor(frame.rgb, cv2.COLOR_RGB2GRAY)
gray_images.append(gray)
# Run calibration
calibrator = IntrinsicCalibrator()
result = calibrator.calibrate(detector, gray_images)
print(f"RMS reprojection error: {result.rms:.3f} px")
print(f"Per-view errors: {result.per_view_errors}")
print(f"Quality gate passed: {result.quality_gate_passed}")
# quality_gate_passed is True when every per-view error is below 0.5 px.
# If False, recapture with more varied poses and better lighting.
if not result.quality_gate_passed:
raise RuntimeError("Calibration quality insufficient — recapture images.")
# Save to disk
CalibrationStore.save_intrinsics(Path("intrinsics.json"), result)
print("Intrinsics saved to intrinsics.json")The quality_gate_passed flag is True when all per-view reprojection errors are below 0.5 px. Use it as a hard gate before saving — a failed quality gate means the calibration data will introduce localization error.
Hand-eye calibration determines the rigid transform between the camera and the robot end-effector frame. You move the robot to diverse poses while detecting the ChArUco board, then solve AX = XB.
import numpy as np
from pathlib import Path
from eyehand import (
CharucoDetector, PoseCollector, HandEyeCalibrator, CalibrationStore
)
from eyehand import pose_to_matrix
# Load intrinsics from Step 1
from eyehand import IntrinsicCalibrationResult
intrinsic_result = CalibrationStore.load_intrinsics(Path("intrinsics.json"))
intrinsics = intrinsic_result.intrinsics
# Use the same board as Step 1
detector = CharucoDetector(
squares_x=5, squares_y=7,
square_length=0.036, marker_length=0.027,
)
collector = PoseCollector()
# Coverage guidance:
# - Collect at least 15 accepted poses
# - Achieve >= 30 deg rotation per principal axis (x, y, z)
# - Avoid near-identical consecutive poses — PoseCollector rejects them automatically
# - Move the robot in all three rotational axes, not just tilt
R_gripper2base = []
t_gripper2base = []
R_target2cam = []
t_target2cam = []
print("Move the robot to diverse poses. Capture loop running...")
while not collector.is_ready():
frame = camera.capture()
fk_pose = robot.get_ee_pose() # Must be T_ee_in_base
import cv2
gray = cv2.cvtColor(frame.rgb, cv2.COLOR_RGB2GRAY)
detection = detector.detect(gray, intrinsics)
if detection.pose is None:
print("Board not detected — reposition.")
continue
# Convert FK pose to 4x4 matrix for PoseCollector
T_ee = pose_to_matrix(fk_pose)
accepted = collector.add(T_ee)
if not accepted:
print("Pose too similar to previous — move the robot more.")
continue
# Gripper-to-base: extract rotation and translation from FK matrix
R_gripper2base.append(T_ee[:3, :3])
t_gripper2base.append(T_ee[:3, 3:4]) # shape (3, 1)
# Target-to-camera: from ChArUco board pose
from eyehand import pose_to_matrix
T_board = pose_to_matrix(detection.pose)
R_target2cam.append(T_board[:3, :3])
t_target2cam.append(T_board[:3, 3:4]) # shape (3, 1)
report = collector.coverage_report()
print(f"Poses: {len(collector)} | Coverage deg — "
f"x:{report['x']:.0f} y:{report['y']:.0f} z:{report['z']:.0f}")
# Run hand-eye calibration with optional ANDREFF cross-check
calibrator = HandEyeCalibrator()
result = calibrator.calibrate(
R_gripper2base, t_gripper2base,
R_target2cam, t_target2cam,
cross_check=True, # Compares PARK and ANDREFF solvers
)
print(f"Method: {result.method}")
print(f"Poses used: {result.num_poses}")
if result.discrepancy_deg is not None:
print(f"PARK vs ANDREFF discrepancy: {result.discrepancy_deg:.2f} deg")
# Discrepancy < 1 deg indicates good solver agreement
CalibrationStore.save_hand_eye(Path("hand_eye.json"), result)
print("Hand-eye calibration saved to hand_eye.json")fk_pose convention: must be T_ee_in_base — the end-effector expressed in the robot base frame (the standard FK result where the transform maps from EE coordinates to base coordinates). If your FK returns T_base_in_ee, invert it first using eyehand.invert_transform.
With both calibration files in hand, create a WorkspaceLocalizer and use it in your runtime loop.
from pathlib import Path
import cv2
from eyehand import (
WorkspaceLocalizer, CalibrationStore, AprilTagDetector
)
# Load calibration once at startup
hand_eye = CalibrationStore.load_hand_eye(Path("hand_eye.json"))
intrinsics = CalibrationStore.load_intrinsics(Path("intrinsics.json")).intrinsics
# Create the localizer — tag_size must match physical tag side length in metres
localizer = WorkspaceLocalizer(
hand_eye=hand_eye,
detector=AprilTagDetector(tag_size=0.05), # 50 mm tags
intrinsics=intrinsics,
)
# Runtime loop
while True:
frame = camera.capture()
fk_pose = robot.get_ee_pose() # T_ee_in_base
result = localizer.localize(frame, fk_pose)
if not result.tag_poses:
print("No tags detected.")
continue
# Filter out geometrically ambiguous detections before using poses
reliable_poses = {
tag_id: pose
for tag_id, pose in result.tag_poses.items()
if tag_id not in result.ambiguous_ids
}
for tag_id, pose in reliable_poses.items():
print(f"Tag {tag_id}: position={pose.position} m (base frame)")The localize() transform chain is:
T_tag_in_base = T_ee_in_base @ T_cam_in_ee @ T_tag_in_cam
T_cam_in_ee is pre-computed from the hand-eye calibration at WorkspaceLocalizer.__init__ time, so only T_ee_in_base (the FK pose) varies per call.
About ambiguous_ids: The IPPE_SQUARE pose solver can produce two geometrically equally plausible solutions for a tag when it is viewed at a shallow angle or is small in the frame. Tags in ambiguous_ids should be filtered out or treated with caution until a better viewing angle is available.
Before deploying, use TouchTestVerifier to validate end-to-end accuracy. The verifier measures the translation error between the robot's ground-truth tool-tip position and the localizer's predicted position.
from eyehand import TouchTestVerifier
verifier = TouchTestVerifier(localizer)
# Move the robot so the tool tip touches each AprilTag in sequence.
# For each touch position, call verify():
verification = verifier.verify(
frame=camera.capture(),
fk_pose=robot.get_ee_pose(),
tag_id=0,
expected_tool_position=robot.get_ee_pose().position, # ground truth
)
print(f"Translation error: {verification.translation_error_m * 1000:.1f} mm")
print(f"Rotation error: {verification.rotation_error_deg:.2f} deg")
print(f"Passed (< 5 mm): {verification.passed}")The default acceptance threshold is 5 mm translation error. If verification fails, recalibrate with more pose diversity and verify your physical measurements are accurate.
No tags detected
- Ensure the image passed to the detector is grayscale (uint8, shape
(H, W)).WorkspaceLocalizer.localizehandles this automatically; directAprilTagDetector.detectcalls require a grayscale array. - Check lighting — avoid strong directional light creating glare on printed tags.
- Verify
tag_sizematches the actual printed tag side length in metres. A wrong value here produces incorrect pose estimates even when detection succeeds.
Positions are mirrored or systematically wrong
- Check the FK convention.
fk_posemust beT_ee_in_base(end-effector in base). If your robot's FK returns the inverse (T_base_in_ee), applyeyehand.invert_transformbefore passing tolocalize(). - Confirm the coordinate frame of your robot matches your expectation — some robots use Z-up, others Y-up.
Large position errors (> 10 mm)
- Recalibrate with more poses (aim for 20+) covering all three rotational axes.
- Re-measure board square size and tag size with calipers — 1 mm measurement error can produce several mm of localization error.
- Check that the ChArUco board is printed flat and not distorted (no warping, inkjet shrinkage, or perspective from a non-flat surface).
- Run
cross_check=TrueinHandEyeCalibrator.calibrateand checkdiscrepancy_deg. Values above 2 deg indicate poor pose diversity.