Skip to content

fix(ios): match Android's preprocessing, version, and permissions #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
blclo wants to merge 1 commit into
base: wildlife-reid
Choose a base branch
from
Open

fix(ios): match Android's preprocessing, version, and permissions #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .husky/pre-commit
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@ fi
if [ -n "$STAGED_SWIFT" ]; then
if command -v swiftlint >/dev/null 2>&1; then
echo "▶ SwiftLint (staged files)..."
echo "$STAGED_SWIFT" | tr '\n' '\0' | xargs -0 swiftlint lint --quiet
# --force-exclude: passing explicit paths otherwise overrides the `excluded`
# list in .swiftlint.yml, so the hook would lint files CI does not (notably
# ios/OffgridMobileTests) and fail on violations no merge gate enforces.
echo "$STAGED_SWIFT" | tr '\n' '\0' | xargs -0 swiftlint lint --quiet --force-exclude
else
echo "⚠️ SwiftLint not installed — skipping Swift lint. Install: brew install swiftlint"
fi
Expand Down
4 changes: 3 additions & 1 deletion docs/setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,9 @@ npm run ios

Apple toolchain compatibility is unresolved: the Gemfile pins `xcodeproj < 1.26.0`, while [Podfile.lock](../ios/Podfile.lock) records CocoaPods 1.17.0. Review that pin before adopting Xcode 16. These steps and a clean iOS build have not been verified on the Windows preparation host; do not treat them as a passing macOS build recipe.

The [Xcode project](../ios/OffgridMobile.xcodeproj/project.pbxproj) leaves `DEVELOPMENT_TEAM` empty. A maintainer must supply their own team and provisioning for real-device builds or archives. Its marketing version is `0.0.58` and does not track the version in [package.json](../package.json) or [android/app/build.gradle](../android/app/build.gradle); iOS release metadata needs separate review.
The [Xcode project](../ios/OffgridMobile.xcodeproj/project.pbxproj) leaves `DEVELOPMENT_TEAM` empty. A maintainer must supply their own team and provisioning for real-device builds or archives. Its `MARKETING_VERSION` and `CURRENT_PROJECT_VERSION` are set by hand to match `versionName` and `versionCode` in [android/app/build.gradle](../android/app/build.gradle); a version bump has to touch both files, and nothing in CI checks that they still agree. `MARKETING_VERSION` carries the full pre-release string, which `CFBundleShortVersionString` does not accept for App Store submission; ad-hoc and AltStore builds are unaffected.

Debug builds install as `org.ganesha.elebook.dev`, matching the Android `applicationIdSuffix`, so a development build sits alongside a tester's field build instead of replacing it. The OAuth redirect scheme stays `org.ganesha.elebook` for both, because it matches on scheme rather than bundle identifier.

## Local checks

Expand Down
101 changes: 86 additions & 15 deletions ios/ImageTensorModule.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,22 +25,21 @@ class ImageTensorModule: NSObject {
rejecter reject: @escaping RCTPromiseRejectBlock
) {
DispatchQueue.global(qos: .userInitiated).async {
guard let image = Self.loadImage(from: uri) else {
guard let source = Self.loadUprightImage(from: uri) else {
reject("IMAGE_ERROR", "Could not load image: \(uri)", nil)
return
}

let targetW = Int(width)
let targetH = Int(height)

guard let resized = Self.resizeImage(image, to: CGSize(width: targetW, height: targetH)),
let cgImage = resized.cgImage else {
guard let resized = Self.progressiveResize(source, width: targetW, height: targetH) else {
reject("IMAGE_ERROR", "Failed to resize image", nil)
return
}

guard let output = Self.extractNchw(
from: cgImage,
from: resized,
width: targetW,
height: targetH,
mean: mean,
Expand Down Expand Up @@ -71,8 +70,7 @@ class ImageTensorModule: NSObject {
rejecter reject: @escaping RCTPromiseRejectBlock
) {
DispatchQueue.global(qos: .userInitiated).async {
guard let image = Self.loadImage(from: uri),
let cgImage = Self.uprightImage(image)?.cgImage else {
guard let cgImage = Self.loadUprightImage(from: uri) else {
reject("IMAGE_ERROR", "Could not load image: \(uri)", nil)
return
}
Expand Down Expand Up @@ -136,14 +134,26 @@ class ImageTensorModule: NSObject {
return nil
}

/// Decode an image and return it on the upright display grid.
///
/// Every consumer in this module -- the detector tensor and the saved crop --
/// goes through here, so they cannot disagree about orientation. Mirrors
/// `loadBitmap` in the Android module.
static func loadUprightImage(from uri: String) -> CGImage? {
guard let image = loadImage(from: uri) else {
return nil
}
return uprightImage(image)?.cgImage
}

/// Redraw an image onto the upright grid described by its EXIF orientation.
///
/// `UIImage.cgImage` is the raw stored buffer and ignores `imageOrientation`,
/// while `imageToTensor` goes through `resizeImage`, whose `draw(in:)` honours
/// it. Cropping the raw buffer therefore cuts from the wrong region for any
/// rotated photo: the on-screen box looks right while the crop MiewID embeds
/// is wrong. Normalising here puts the crop on the same grid as the detector,
/// the overlay, and every EXIF-aware viewer.
/// while every EXIF-aware viewer applies it. Cropping or resampling the raw
/// buffer therefore works from the wrong region for any rotated photo: the
/// on-screen box looks right while the crop MiewID embeds is wrong.
/// Normalising here puts the tensor and the crop on the same grid as the
/// detector, the overlay, and every EXIF-aware viewer.
///
/// Returns the input unchanged when it is already upright.
static func uprightImage(_ image: UIImage) -> UIImage? {
Expand All @@ -159,11 +169,72 @@ class ImageTensorModule: NSObject {
}
}

static func resizeImage(_ image: UIImage, to size: CGSize) -> UIImage? {
let renderer = UIGraphicsImageRenderer(size: size)
return renderer.image { _ in
image.draw(in: CGRect(origin: .zero, size: size))
/// Resize toward (targetWidth, targetHeight) via repeated 2x downscales
/// before the final step, instead of one large single-shot resize.
///
/// Resampling is only accurate for moderate size reductions. For a large
/// reduction in one step (e.g. a multi-megapixel photo down to 440x440 --
/// 10x or more per axis), high-frequency detail gets folded into different
/// values instead of being averaged away, unlike PIL/torchvision's `Resize`
/// (used by the Python reference pipeline), which applies antialiasing before
/// subsampling. Repeated halving approximates that antialiasing with a
/// box/mipmap-style filter chain, so no individual step exceeds 2x.
///
/// This mirrors `progressiveResize` in the Android module, which was verified
/// against Project Ganesha's golden on-device parity test (E13-4). The
/// structure matches; the per-step filter still does not (Android resamples
/// bilinearly via `createScaledBitmap`, this resamples at `.high`), so this
/// buys a shared algorithm, not byte parity.
///
/// What it definitely fixes is device dependence. The previous single-shot
/// path resampled through `UIGraphicsImageRenderer`'s default format, whose
/// scale is the screen scale, so the same photo yielded a different tensor on
/// a 2x and a 3x iPhone -- measured up to 49/255 on a channel for a small
/// crop. See kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md.
static func progressiveResize(_ source: CGImage, width targetWidth: Int, height targetHeight: Int) -> CGImage? {
guard targetWidth > 0, targetHeight > 0 else {
return nil
}
if source.width == targetWidth && source.height == targetHeight {
return source
}

var current = source
while current.width > targetWidth * 2 && current.height > targetHeight * 2 {
let nextWidth = max(targetWidth, current.width / 2)
let nextHeight = max(targetHeight, current.height / 2)
guard let next = redraw(current, width: nextWidth, height: nextHeight) else {
return nil
}
current = next
}

return redraw(current, width: targetWidth, height: targetHeight)
}

/// Resample into an exactly `width` x `height` pixel buffer.
///
/// Deliberately not `UIGraphicsImageRenderer`, whose default format uses the
/// screen scale: on a 3x device that renders a 440x440 request into a
/// 1320x1320 buffer, so the halving chain above would measure the wrong
/// dimensions and `extractNchw` would resample the oversized result a second
/// time. Sizing the context in pixels keeps the chain honest.
private static func redraw(_ cgImage: CGImage, width: Int, height: Int) -> CGImage? {
guard let colorSpace = CGColorSpace(name: CGColorSpace.sRGB),
let context = CGContext(
data: nil,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: 4 * width,
space: colorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue
) else {
return nil
}
context.interpolationQuality = .high
context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height))
return context.makeImage()
}

static func extractNchw(
Expand Down
10 changes: 5 additions & 5 deletions ios/OffgridMobile.xcodeproj/project.pbxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,7 @@
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
CLANG_ENABLE_MODULES = YES;
CURRENT_PROJECT_VERSION = 1771585509;
CURRENT_PROJECT_VERSION = 1787551547;
DEVELOPMENT_TEAM = "";
ENABLE_BITCODE = NO;
INFOPLIST_FILE = OffgridMobile/Info.plist;
Expand All @@ -425,13 +425,13 @@
"$(inherited)",
"@executable_path/Frameworks",
);
MARKETING_VERSION = 0.0.58;
MARKETING_VERSION = "0.1.0-field.6";
OTHER_LDFLAGS = (
"$(inherited)",
"-ObjC",
"-lc++",
);
PRODUCT_BUNDLE_IDENTIFIER = org.ganesha.elebook;
PRODUCT_BUNDLE_IDENTIFIER = org.ganesha.elebook.dev;
PRODUCT_NAME = OffgridMobile;
SWIFT_OBJC_BRIDGING_HEADER = "OffgridMobile/OffgridMobile-Bridging-Header.h";
SWIFT_OPTIMIZATION_LEVEL = "-Onone";
Expand All @@ -446,7 +446,7 @@
buildSettings = {
ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
CLANG_ENABLE_MODULES = YES;
CURRENT_PROJECT_VERSION = 1771585509;
CURRENT_PROJECT_VERSION = 1787551547;
DEVELOPMENT_TEAM = "";
INFOPLIST_FILE = OffgridMobile/Info.plist;
INFOPLIST_KEY_CFBundleDisplayName = EleBook;
Expand All @@ -456,7 +456,7 @@
"$(inherited)",
"@executable_path/Frameworks",
);
MARKETING_VERSION = 0.0.58;
MARKETING_VERSION = "0.1.0-field.6";
OTHER_LDFLAGS = (
"$(inherited)",
"-ObjC",
Expand Down
8 changes: 2 additions & 6 deletions ios/OffgridMobile/Info.plist
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleName</key>
<string>$(PRODUCT_NAME)</string>
<string>EleBook</string>
<key>CFBundleURLTypes</key>
<array>
<dict>
Expand Down Expand Up @@ -43,12 +43,8 @@
<string>This app needs access to your camera to photograph wildlife for on-device identification.</string>
<key>NSLocationWhenInUseUsageDescription</key>
<string>This app uses your location to tag observations with GPS coordinates. Location is optional and only captured while you are recording an observation.</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app needs access to your microphone for voice-to-text transcription using Whisper.</string>
<key>NSPhotoLibraryUsageDescription</key>
<string>This app needs access to your photo library to attach images to conversations.</string>
<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses on-device speech recognition to transcribe voice input.</string>
<string>This app needs access to your photo library to import wildlife photographs for on-device identification.</string>
<key>RCTNewArchEnabled</key>
<true/>
<key>UIAppFonts</key>
Expand Down
10 changes: 1 addition & 9 deletions ios/OffgridMobile/LaunchScreen.storyboard
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,11 @@
<constraint firstAttribute="height" constant="128" id="hgt-Lg-pK1"/>
</constraints>
</imageView>
<label opaque="NO" clipsSubviews="YES" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" text="Off Grid" textAlignment="center" lineBreakMode="middleTruncation" baselineAdjustment="alignBaselines" minimumFontSize="18" translatesAutoresizingMaskIntoConstraints="NO" id="GJd-Yh-RWb">
<rect key="frame" x="0.0" y="367" width="375" height="43"/>
<fontDescription key="fontDescription" type="boldSystem" pointSize="36"/>
<nil key="highlightedColor"/>
</label>
</subviews>
<color key="backgroundColor" systemColor="systemBackgroundColor" cocoaTouchSystemColor="whiteColor"/>
<constraints>
<constraint firstItem="Lgp-K1-zXR" firstAttribute="centerX" secondItem="Ze5-6b-2t3" secondAttribute="centerX" id="cxL-gp-K1z"/>
<constraint firstItem="Lgp-K1-zXR" firstAttribute="centerY" secondItem="Ze5-6b-2t3" secondAttribute="centerY" constant="-40" id="cyL-gp-K1z"/>
<constraint firstItem="GJd-Yh-RWb" firstAttribute="top" secondItem="Lgp-K1-zXR" secondAttribute="bottom" constant="16" id="tpG-Jd-Yh1"/>
<constraint firstItem="GJd-Yh-RWb" firstAttribute="leading" secondItem="Bcu-3y-fUS" secondAttribute="leading" id="ldG-Jd-Yh1"/>
<constraint firstItem="GJd-Yh-RWb" firstAttribute="trailing" secondItem="Bcu-3y-fUS" secondAttribute="trailing" id="trG-Jd-Yh1"/>
<constraint firstItem="Lgp-K1-zXR" firstAttribute="centerY" secondItem="Ze5-6b-2t3" secondAttribute="centerY" id="cyL-gp-K1z"/>
</constraints>
<viewLayoutGuide key="safeArea" id="Bcu-3y-fUS"/>
</view>
Expand Down
101 changes: 97 additions & 4 deletions ios/OffgridMobileTests/OffgridMobileTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -944,10 +944,11 @@ final class ImageTensorModuleTests: XCTestCase {

// MARK: - EXIF orientation parity
//
// `imageToTensor` goes through `resizeImage`, whose `draw(in:)` applies
// `imageOrientation`; `cropImage` used the raw `cgImage`, which does not.
// For any rotated photo that put the detector and the crop on different
// grids, so MiewID embedded the wrong pixels while the overlay looked fine.
// Both bridge methods now decode through `loadUprightImage`, which applies
// `imageOrientation` once. Before that, `cropImage` read the raw `cgImage`,
// which does not. For any rotated photo that put the detector and the crop
// on different grids, so MiewID embedded the wrong pixels while the overlay
// looked fine.

/// A renderer pinned to scale 1, so the backing buffer is exactly the
/// requested pixel size. The default format uses the screen scale, which
Expand Down Expand Up @@ -1162,6 +1163,98 @@ final class ImageTensorModuleTests: XCTestCase {
}
}

// MARK: - Progressive resize parity
//
// Android reduces toward the target in repeated 2x steps so a large
// downscale is antialiased rather than aliased, and that chain was pinned by
// Project Ganesha's golden on-device parity test (E13-4). iOS resized in one
// shot through a screen-scaled renderer, so the same photo produced a
// different tensor -- and therefore different MiewID candidates -- per
// platform.

/// Alternating 1px stripes: the finest detail a resampler can carry, so a
/// large reduction either averages them to mid-grey or aliases toward the
/// extremes.
private func makeStripedImage(width: Int, height: Int) -> UIImage {
return unscaledRenderer(width: width, height: height).image { ctx in
for column in 0..<width {
(column % 2 == 0 ? UIColor.black : UIColor.white).setFill()
ctx.fill(CGRect(x: column, y: 0, width: 1, height: height))
}
}
}

func testProgressiveResizeReturnsExactTargetPixelDimensions() throws {
// The old renderer used UIGraphicsImageRenderer's default format, whose
// scale is the screen scale, so a 50x50 request produced a 100x100 or
// 150x150 buffer on a 2x or 3x simulator.
let source = try XCTUnwrap(makeStripedImage(width: 800, height: 400).cgImage)
let resized = try XCTUnwrap(ImageTensorModule.progressiveResize(source, width: 50, height: 50))

XCTAssertEqual(resized.width, 50)
XCTAssertEqual(resized.height, 50)
}

func testProgressiveResizeReturnsSourceUntouchedAtTargetSize() throws {
let source = try XCTUnwrap(makeSplitImage(width: 4, height: 2).cgImage)
let resized = try XCTUnwrap(ImageTensorModule.progressiveResize(source, width: 4, height: 2))

XCTAssertEqual(resized.width, 4)
XCTAssertEqual(resized.height, 2)
}

func testProgressiveResizeRejectsNonPositiveTarget() throws {
let source = try XCTUnwrap(makeSplitImage(width: 4, height: 2).cgImage)

XCTAssertNil(ImageTensorModule.progressiveResize(source, width: 0, height: 8))
XCTAssertNil(ImageTensorModule.progressiveResize(source, width: 8, height: -1))
}

func testProgressiveResizeAveragesFineDetailRatherThanAliasing() throws {
// 640 -> 20 is a 32x reduction per axis. Every output pixel covers an equal
// number of black and white columns, so the area average is mid-grey.
let source = try XCTUnwrap(makeStripedImage(width: 640, height: 640).cgImage)
let resized = try XCTUnwrap(ImageTensorModule.progressiveResize(source, width: 20, height: 20))
let tensor = try XCTUnwrap(ImageTensorModule.extractNchw(
from: resized, width: 20, height: 20,
mean: [0, 0, 0], std: [1, 1, 1], scale: 1.0, bgr: false
))

for (index, value) in tensor.enumerated() {
XCTAssertEqual(value, 127.5, accuracy: 24.0, "channel sample \(index) aliased")
}
}

func testImageToTensorAveragesFineDetailThroughTheBridge() throws {
let url = FileManager.default.temporaryDirectory
.appendingPathComponent(UUID().uuidString + ".png")
let image = makeStripedImage(width: 640, height: 640)
try XCTUnwrap(image.pngData()).write(to: url)
defer { try? FileManager.default.removeItem(at: url) }

let completed = expectation(description: "tensor")
var tensor: [Double]?
module.imageToTensor(
url.path, width: 20, height: 20, mean: [0, 0, 0], std: [1, 1, 1],
scale: 1.0, channelOrder: "RGB",
resolver: { value in
tensor = value as? [Double]
completed.fulfill()
},
rejecter: { _, message, _ in
XCTFail("striped tensor rejected: \(message ?? "unknown")")
completed.fulfill()
}
)
waitForExpectations(timeout: 10)

let values = try XCTUnwrap(tensor)
XCTAssertEqual(values.count, 3 * 20 * 20)
for (index, value) in values.enumerated() {
XCTAssertEqual(value, 127.5, accuracy: 24.0, "channel sample \(index) aliased")
}
}

private func createTestImage(width: Int, height: Int, color: UIColor) -> UIImage {
let size = CGSize(width: width, height: height)
let renderer = UIGraphicsImageRenderer(size: size)
Expand Down
Loading
Loading