From 033040122db14db932531abfe228c9eb49045778 Mon Sep 17 00:00:00 2001 From: Carolina Lopez <99345307+blclo@users.noreply.github.com> Date: Sun, 13 Sep 2026 13:36:58 +0200 Subject: [PATCH] fix(ios): match Android's preprocessing, version, and permissions The shared TypeScript already ran identically on both platforms; the drift was all in the native layer. Preprocessing. Android reduces toward the target in repeated 2x steps (progressiveResize, pinned by the E13-4 golden on-device parity test). iOS resized in one shot through UIGraphicsImageRenderer's default format, whose scale is the screen scale, so a 440x440 request allocated a 1320x1320 buffer on a 3x device and 880x880 on a 2x one -- which extractNchw then resampled again. The tensor depended on which iPhone computed it: measured up to 49/255 on a channel between a 2x and a 3x device for a small crop. progressiveResize now mirrors the Kotlin algorithm and draws into an explicit pixel-sized CGContext. This buys a shared algorithm, not byte parity -- Android resamples bilinearly via createScaledBitmap, Core Graphics at .high -- and no MiewID candidate-ordering impact was measured. Numbers in kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md. Both bridge methods now decode through one loadUprightImage helper, mirroring Android's loadBitmap, so tensor and crop cannot disagree about EXIF orientation. Identity. MARKETING_VERSION was 0.0.58 while Android shipped 0.1.0-field.6; the field.4/5/6 bumps each touched only build.gradle. Version and build number now match Android. Debug takes the .dev bundle suffix Android already uses, so a dev build no longer replaces a tester's field build. The launch screen no longer reads "Off Grid", and CFBundleName is EleBook rather than the Xcode target name. Permissions. Info.plist asked for the microphone "using Whisper" and for speech recognition, and described the photo library as being for conversations. No audio dependency remains and the picker is photo-only, so both stale requests are gone and the photo string describes the app. Hook. The pre-commit SwiftLint step passed explicit paths, which overrides .swiftlint.yml's `excluded` list, so it failed on 8 pre-existing violations in ios/OffgridMobileTests that no merge gate enforces. It never fired before because the recent iOS work was done on hosts without SwiftLint installed. --force-exclude makes the hook agree with `npm run lint:ios`. Co-Authored-By: Claude Opus 5 --- .husky/pre-commit | 5 +- docs/setup.md | 4 +- ios/ImageTensorModule.swift | 101 +++++++++++++++--- ios/OffgridMobile.xcodeproj/project.pbxproj | 10 +- ios/OffgridMobile/Info.plist | 8 +- ios/OffgridMobile/LaunchScreen.storyboard | 10 +- .../OffgridMobileTests.swift | 101 +++++++++++++++++- ...026-09-13-ios-resize-parity-measurement.md | 37 +++++++ 8 files changed, 235 insertions(+), 41 deletions(-) create mode 100644 kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md diff --git a/.husky/pre-commit b/.husky/pre-commit index 1c6b10daa..b057e869b 100755 --- a/.husky/pre-commit +++ b/.husky/pre-commit @@ -21,7 +21,10 @@ fi if [ -n "$STAGED_SWIFT" ]; then if command -v swiftlint >/dev/null 2>&1; then echo "▶ SwiftLint (staged files)..." - echo "$STAGED_SWIFT" | tr '\n' '\0' | xargs -0 swiftlint lint --quiet + # --force-exclude: passing explicit paths otherwise overrides the `excluded` + # list in .swiftlint.yml, so the hook would lint files CI does not (notably + # ios/OffgridMobileTests) and fail on violations no merge gate enforces. + echo "$STAGED_SWIFT" | tr '\n' '\0' | xargs -0 swiftlint lint --quiet --force-exclude else echo "⚠️ SwiftLint not installed — skipping Swift lint. Install: brew install swiftlint" fi diff --git a/docs/setup.md b/docs/setup.md index 98bde0925..cddbf2763 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -115,7 +115,9 @@ npm run ios Apple toolchain compatibility is unresolved: the Gemfile pins `xcodeproj < 1.26.0`, while [Podfile.lock](../ios/Podfile.lock) records CocoaPods 1.17.0. Review that pin before adopting Xcode 16. These steps and a clean iOS build have not been verified on the Windows preparation host; do not treat them as a passing macOS build recipe. -The [Xcode project](../ios/OffgridMobile.xcodeproj/project.pbxproj) leaves `DEVELOPMENT_TEAM` empty. A maintainer must supply their own team and provisioning for real-device builds or archives. Its marketing version is `0.0.58` and does not track the version in [package.json](../package.json) or [android/app/build.gradle](../android/app/build.gradle); iOS release metadata needs separate review. +The [Xcode project](../ios/OffgridMobile.xcodeproj/project.pbxproj) leaves `DEVELOPMENT_TEAM` empty. A maintainer must supply their own team and provisioning for real-device builds or archives. Its `MARKETING_VERSION` and `CURRENT_PROJECT_VERSION` are set by hand to match `versionName` and `versionCode` in [android/app/build.gradle](../android/app/build.gradle); a version bump has to touch both files, and nothing in CI checks that they still agree. `MARKETING_VERSION` carries the full pre-release string, which `CFBundleShortVersionString` does not accept for App Store submission; ad-hoc and AltStore builds are unaffected. + +Debug builds install as `org.ganesha.elebook.dev`, matching the Android `applicationIdSuffix`, so a development build sits alongside a tester's field build instead of replacing it. The OAuth redirect scheme stays `org.ganesha.elebook` for both, because it matches on scheme rather than bundle identifier. ## Local checks diff --git a/ios/ImageTensorModule.swift b/ios/ImageTensorModule.swift index ba5c81637..5aedddd55 100644 --- a/ios/ImageTensorModule.swift +++ b/ios/ImageTensorModule.swift @@ -25,7 +25,7 @@ class ImageTensorModule: NSObject { rejecter reject: @escaping RCTPromiseRejectBlock ) { DispatchQueue.global(qos: .userInitiated).async { - guard let image = Self.loadImage(from: uri) else { + guard let source = Self.loadUprightImage(from: uri) else { reject("IMAGE_ERROR", "Could not load image: \(uri)", nil) return } @@ -33,14 +33,13 @@ class ImageTensorModule: NSObject { let targetW = Int(width) let targetH = Int(height) - guard let resized = Self.resizeImage(image, to: CGSize(width: targetW, height: targetH)), - let cgImage = resized.cgImage else { + guard let resized = Self.progressiveResize(source, width: targetW, height: targetH) else { reject("IMAGE_ERROR", "Failed to resize image", nil) return } guard let output = Self.extractNchw( - from: cgImage, + from: resized, width: targetW, height: targetH, mean: mean, @@ -71,8 +70,7 @@ class ImageTensorModule: NSObject { rejecter reject: @escaping RCTPromiseRejectBlock ) { DispatchQueue.global(qos: .userInitiated).async { - guard let image = Self.loadImage(from: uri), - let cgImage = Self.uprightImage(image)?.cgImage else { + guard let cgImage = Self.loadUprightImage(from: uri) else { reject("IMAGE_ERROR", "Could not load image: \(uri)", nil) return } @@ -136,14 +134,26 @@ class ImageTensorModule: NSObject { return nil } + /// Decode an image and return it on the upright display grid. + /// + /// Every consumer in this module -- the detector tensor and the saved crop -- + /// goes through here, so they cannot disagree about orientation. Mirrors + /// `loadBitmap` in the Android module. + static func loadUprightImage(from uri: String) -> CGImage? { + guard let image = loadImage(from: uri) else { + return nil + } + return uprightImage(image)?.cgImage + } + /// Redraw an image onto the upright grid described by its EXIF orientation. /// /// `UIImage.cgImage` is the raw stored buffer and ignores `imageOrientation`, - /// while `imageToTensor` goes through `resizeImage`, whose `draw(in:)` honours - /// it. Cropping the raw buffer therefore cuts from the wrong region for any - /// rotated photo: the on-screen box looks right while the crop MiewID embeds - /// is wrong. Normalising here puts the crop on the same grid as the detector, - /// the overlay, and every EXIF-aware viewer. + /// while every EXIF-aware viewer applies it. Cropping or resampling the raw + /// buffer therefore works from the wrong region for any rotated photo: the + /// on-screen box looks right while the crop MiewID embeds is wrong. + /// Normalising here puts the tensor and the crop on the same grid as the + /// detector, the overlay, and every EXIF-aware viewer. /// /// Returns the input unchanged when it is already upright. static func uprightImage(_ image: UIImage) -> UIImage? { @@ -159,11 +169,72 @@ class ImageTensorModule: NSObject { } } - static func resizeImage(_ image: UIImage, to size: CGSize) -> UIImage? { - let renderer = UIGraphicsImageRenderer(size: size) - return renderer.image { _ in - image.draw(in: CGRect(origin: .zero, size: size)) + /// Resize toward (targetWidth, targetHeight) via repeated 2x downscales + /// before the final step, instead of one large single-shot resize. + /// + /// Resampling is only accurate for moderate size reductions. For a large + /// reduction in one step (e.g. a multi-megapixel photo down to 440x440 -- + /// 10x or more per axis), high-frequency detail gets folded into different + /// values instead of being averaged away, unlike PIL/torchvision's `Resize` + /// (used by the Python reference pipeline), which applies antialiasing before + /// subsampling. Repeated halving approximates that antialiasing with a + /// box/mipmap-style filter chain, so no individual step exceeds 2x. + /// + /// This mirrors `progressiveResize` in the Android module, which was verified + /// against Project Ganesha's golden on-device parity test (E13-4). The + /// structure matches; the per-step filter still does not (Android resamples + /// bilinearly via `createScaledBitmap`, this resamples at `.high`), so this + /// buys a shared algorithm, not byte parity. + /// + /// What it definitely fixes is device dependence. The previous single-shot + /// path resampled through `UIGraphicsImageRenderer`'s default format, whose + /// scale is the screen scale, so the same photo yielded a different tensor on + /// a 2x and a 3x iPhone -- measured up to 49/255 on a channel for a small + /// crop. See kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md. + static func progressiveResize(_ source: CGImage, width targetWidth: Int, height targetHeight: Int) -> CGImage? { + guard targetWidth > 0, targetHeight > 0 else { + return nil + } + if source.width == targetWidth && source.height == targetHeight { + return source + } + + var current = source + while current.width > targetWidth * 2 && current.height > targetHeight * 2 { + let nextWidth = max(targetWidth, current.width / 2) + let nextHeight = max(targetHeight, current.height / 2) + guard let next = redraw(current, width: nextWidth, height: nextHeight) else { + return nil + } + current = next + } + + return redraw(current, width: targetWidth, height: targetHeight) + } + + /// Resample into an exactly `width` x `height` pixel buffer. + /// + /// Deliberately not `UIGraphicsImageRenderer`, whose default format uses the + /// screen scale: on a 3x device that renders a 440x440 request into a + /// 1320x1320 buffer, so the halving chain above would measure the wrong + /// dimensions and `extractNchw` would resample the oversized result a second + /// time. Sizing the context in pixels keeps the chain honest. + private static func redraw(_ cgImage: CGImage, width: Int, height: Int) -> CGImage? { + guard let colorSpace = CGColorSpace(name: CGColorSpace.sRGB), + let context = CGContext( + data: nil, + width: width, + height: height, + bitsPerComponent: 8, + bytesPerRow: 4 * width, + space: colorSpace, + bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue + ) else { + return nil } + context.interpolationQuality = .high + context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height)) + return context.makeImage() } static func extractNchw( diff --git a/ios/OffgridMobile.xcodeproj/project.pbxproj b/ios/OffgridMobile.xcodeproj/project.pbxproj index 6e9ddf6c9..1a5c78fe4 100644 --- a/ios/OffgridMobile.xcodeproj/project.pbxproj +++ b/ios/OffgridMobile.xcodeproj/project.pbxproj @@ -414,7 +414,7 @@ buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; CLANG_ENABLE_MODULES = YES; - CURRENT_PROJECT_VERSION = 1771585509; + CURRENT_PROJECT_VERSION = 1787551547; DEVELOPMENT_TEAM = ""; ENABLE_BITCODE = NO; INFOPLIST_FILE = OffgridMobile/Info.plist; @@ -425,13 +425,13 @@ "$(inherited)", "@executable_path/Frameworks", ); - MARKETING_VERSION = 0.0.58; + MARKETING_VERSION = "0.1.0-field.6"; OTHER_LDFLAGS = ( "$(inherited)", "-ObjC", "-lc++", ); - PRODUCT_BUNDLE_IDENTIFIER = org.ganesha.elebook; + PRODUCT_BUNDLE_IDENTIFIER = org.ganesha.elebook.dev; PRODUCT_NAME = OffgridMobile; SWIFT_OBJC_BRIDGING_HEADER = "OffgridMobile/OffgridMobile-Bridging-Header.h"; SWIFT_OPTIMIZATION_LEVEL = "-Onone"; @@ -446,7 +446,7 @@ buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; CLANG_ENABLE_MODULES = YES; - CURRENT_PROJECT_VERSION = 1771585509; + CURRENT_PROJECT_VERSION = 1787551547; DEVELOPMENT_TEAM = ""; INFOPLIST_FILE = OffgridMobile/Info.plist; INFOPLIST_KEY_CFBundleDisplayName = EleBook; @@ -456,7 +456,7 @@ "$(inherited)", "@executable_path/Frameworks", ); - MARKETING_VERSION = 0.0.58; + MARKETING_VERSION = "0.1.0-field.6"; OTHER_LDFLAGS = ( "$(inherited)", "-ObjC", diff --git a/ios/OffgridMobile/Info.plist b/ios/OffgridMobile/Info.plist index 55b89503c..80dee96e8 100644 --- a/ios/OffgridMobile/Info.plist +++ b/ios/OffgridMobile/Info.plist @@ -15,7 +15,7 @@ CFBundleInfoDictionaryVersion 6.0 CFBundleName - $(PRODUCT_NAME) + EleBook CFBundleURLTypes @@ -43,12 +43,8 @@ This app needs access to your camera to photograph wildlife for on-device identification. NSLocationWhenInUseUsageDescription This app uses your location to tag observations with GPS coordinates. Location is optional and only captured while you are recording an observation. - NSMicrophoneUsageDescription - This app needs access to your microphone for voice-to-text transcription using Whisper. NSPhotoLibraryUsageDescription - This app needs access to your photo library to attach images to conversations. - NSSpeechRecognitionUsageDescription - This app uses on-device speech recognition to transcribe voice input. + This app needs access to your photo library to import wildlife photographs for on-device identification. RCTNewArchEnabled UIAppFonts diff --git a/ios/OffgridMobile/LaunchScreen.storyboard b/ios/OffgridMobile/LaunchScreen.storyboard index fdc229281..d6efcc2b3 100644 --- a/ios/OffgridMobile/LaunchScreen.storyboard +++ b/ios/OffgridMobile/LaunchScreen.storyboard @@ -23,19 +23,11 @@ - - - - - + diff --git a/ios/OffgridMobileTests/OffgridMobileTests.swift b/ios/OffgridMobileTests/OffgridMobileTests.swift index 69af6c293..e99fab3fe 100644 --- a/ios/OffgridMobileTests/OffgridMobileTests.swift +++ b/ios/OffgridMobileTests/OffgridMobileTests.swift @@ -944,10 +944,11 @@ final class ImageTensorModuleTests: XCTestCase { // MARK: - EXIF orientation parity // - // `imageToTensor` goes through `resizeImage`, whose `draw(in:)` applies - // `imageOrientation`; `cropImage` used the raw `cgImage`, which does not. - // For any rotated photo that put the detector and the crop on different - // grids, so MiewID embedded the wrong pixels while the overlay looked fine. + // Both bridge methods now decode through `loadUprightImage`, which applies + // `imageOrientation` once. Before that, `cropImage` read the raw `cgImage`, + // which does not. For any rotated photo that put the detector and the crop + // on different grids, so MiewID embedded the wrong pixels while the overlay + // looked fine. /// A renderer pinned to scale 1, so the backing buffer is exactly the /// requested pixel size. The default format uses the screen scale, which @@ -1162,6 +1163,98 @@ final class ImageTensorModuleTests: XCTestCase { } } + // MARK: - Progressive resize parity + // + // Android reduces toward the target in repeated 2x steps so a large + // downscale is antialiased rather than aliased, and that chain was pinned by + // Project Ganesha's golden on-device parity test (E13-4). iOS resized in one + // shot through a screen-scaled renderer, so the same photo produced a + // different tensor -- and therefore different MiewID candidates -- per + // platform. + + /// Alternating 1px stripes: the finest detail a resampler can carry, so a + /// large reduction either averages them to mid-grey or aliases toward the + /// extremes. + private func makeStripedImage(width: Int, height: Int) -> UIImage { + return unscaledRenderer(width: width, height: height).image { ctx in + for column in 0.. 20 is a 32x reduction per axis. Every output pixel covers an equal + // number of black and white columns, so the area average is mid-grey. + let source = try XCTUnwrap(makeStripedImage(width: 640, height: 640).cgImage) + let resized = try XCTUnwrap(ImageTensorModule.progressiveResize(source, width: 20, height: 20)) + let tensor = try XCTUnwrap(ImageTensorModule.extractNchw( + from: resized, width: 20, height: 20, + mean: [0, 0, 0], std: [1, 1, 1], scale: 1.0, bgr: false + )) + + for (index, value) in tensor.enumerated() { + XCTAssertEqual(value, 127.5, accuracy: 24.0, "channel sample \(index) aliased") + } + } + + func testImageToTensorAveragesFineDetailThroughTheBridge() throws { + let url = FileManager.default.temporaryDirectory + .appendingPathComponent(UUID().uuidString + ".png") + let image = makeStripedImage(width: 640, height: 640) + try XCTUnwrap(image.pngData()).write(to: url) + defer { try? FileManager.default.removeItem(at: url) } + + let completed = expectation(description: "tensor") + var tensor: [Double]? + module.imageToTensor( + url.path, width: 20, height: 20, mean: [0, 0, 0], std: [1, 1, 1], + scale: 1.0, channelOrder: "RGB", + resolver: { value in + tensor = value as? [Double] + completed.fulfill() + }, + rejecter: { _, message, _ in + XCTFail("striped tensor rejected: \(message ?? "unknown")") + completed.fulfill() + } + ) + waitForExpectations(timeout: 10) + + let values = try XCTUnwrap(tensor) + XCTAssertEqual(values.count, 3 * 20 * 20) + for (index, value) in values.enumerated() { + XCTAssertEqual(value, 127.5, accuracy: 24.0, "channel sample \(index) aliased") + } + } + private func createTestImage(width: Int, height: Int, color: UIColor) -> UIImage { let size = CGSize(width: width, height: height) let renderer = UIGraphicsImageRenderer(size: size) diff --git a/kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md b/kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md new file mode 100644 index 000000000..f65dcb0d3 --- /dev/null +++ b/kb/wildlife-reid-mobile/outputs/reports/2026-09-13-ios-resize-parity-measurement.md @@ -0,0 +1,37 @@ +# iOS preprocessing parity — measured before/after + +**Date:** 2026-09-13 +**Subject:** `ios/ImageTensorModule.swift` resize path vs `android/.../ImageTensorModule.kt` `progressiveResize` +**Method:** standalone Swift binary run under `xcrun simctl spawn` on an iPhone 17 Pro simulator, exercising the old and new resampling paths on identical in-memory fixtures. Not a device test and not a MiewID test. + +## What was wrong + +Android reduces toward the target in repeated 2x steps (`progressiveResize`, pinned by the E13-4 golden on-device parity test). iOS did a single `UIImage.draw(in:)` through a `UIGraphicsImageRenderer` built with the **default** format, whose `scale` is `UIScreen.main.scale`. Two consequences: + +1. A 440x440 request allocated a 440·scale buffer — measured at **1320x1320 on a 3x device, 880x880 on a 2x device** — which `extractNchw` then resampled a second time down to 440. +2. Because the intermediate size came from the screen, **the tensor depended on which iPhone ran it**. + +## Measurements + +Detail-dense diagonal-texture fixture, reduced to 440x440. Values are absolute per-channel differences out of 255. + +| Source | old@2x vs old@3x | new vs old@2x | new vs old@3x | +| --- | --- | --- | --- | +| 4032x3024 (12MP full frame) | max 2, mean 0.23 | max 7, mean 0.31 | max 5, mean 0.29 | +| 2048x1536 (large crop) | max 6, mean 0.48 | max 4, mean 0.41 | max 6, mean 0.43 | +| 1500x1200 (mid crop) | max 26, mean 1.69 | max 9, mean 1.63 | max 27, mean 2.17 | +| 880x700 (small crop) | max 49, mean 3.57 | max 33, mean 2.62 | max 41, mean 3.84 | + +The first column is the device-dependence bug on its own: the same photo, the same build, two different iPhones, up to 49/255 apart on a channel. The new path removes it — output is a function of the image alone. + +For a 1760x1760 source the old 3x path coincidentally matched the new chain at 2x exactly (max 0), because 1760 -> 880 -> 440 *is* the halving chain. The coincidence does not hold at other sizes, which is the point: the old behaviour was an accident of screen scale. + +## What this does not establish + +- **Not byte parity with Android.** The chain structure now matches, but the per-step filter does not: Android uses `Bitmap.createScaledBitmap(filter = true)` (bilinear), iOS uses `CGContext` at `.high` interpolation. Same algorithm, different kernel. +- **No MiewID impact measured.** Differences of a few units out of 255 may or may not reorder candidates. Establishing that needs the golden batch run on both platforms against the same images — the E13-4 equivalent for iOS, which does not exist yet. +- Synthetic high-frequency fixtures are a worst case. Real fur and skin will differ by less. + +## Follow-up worth doing + +An iOS counterpart to the E13-4 on-device parity test, comparing embeddings for a fixed image set against the Python reference, is the only thing that would turn "the algorithm matches" into "the results match".