fix: put detector, overlay, and crop on one EXIF-upright grid - #35
Conversation
BitmapFactory and UIImage.cgImage return the stored sensor buffer and ignore the EXIF orientation tag, while React Native's <Image> and every EXIF-aware viewer apply it. Detection boxes were normalized against one grid and drawn on another: transposed overlays on Android, and on iOS a correct-looking overlay over crops cut from the wrong region, so MiewID embedded the wrong pixels and match scores degraded silently. Android: apply the orientation tag once in loadBitmap, so imageToTensor and cropImage both work from the display grid. iOS: normalize orientation before cropping, matching what imageToTensor already did implicitly through resizeImage. Adds 11 Robolectric tests covering all eight EXIF orientations plus two matchability guards asserting that the MiewID input tensor and the crop region are identical whether a photo is stored upright or rotated. Robolectric's legacy bitmap shadow does not transform pixels for matrix-backed createBitmap, so these run under GraphicsMode.NATIVE in their own class. The iOS tests are written but unverified: this host has no xcodebuild and CI has no iOS test job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both gaps came out of a Codex review of the fix. The orientation tests all called applyExifOrientation and uprightImage directly, so removing the call from loadBitmap or cropImage would have left every one of them green. Adds an end-to-end test through the cropImage bridge method using a real EXIF-tagged JPEG. Verified by unwiring loadBitmap: exactly that test fails, and it passes again once restored. It goes through cropImage rather than imageToTensor because imageToTensor resolves a WritableNativeArray, which needs React Native's JNI that Robolectric does not load. The iOS fixtures used UIGraphicsImageRenderer's default screen scale while rewrapping the buffer at scale 1, so a "4x2" image is an 8x4 or 12x6 buffer on a 2x or 3x simulator and the dimension assertions would fail there despite correct production code. Both fixtures now render at an explicit scale of 1. Android: 12 orientation tests pass, whole suite green. The iOS tests remain unverified on this host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the verdict, the two Major test gaps and how each was closed, the two Minor items deferred with reasoning, and the open gaps -- chiefly that the iOS tests have never run, because this host has no xcodebuild and CI has no iOS test job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex second-opinion reviewRan a scoped Codex review over the Android transform, the iOS normalization, the It independently derived the source-to-upright pixel mapping for all eight EXIF Two Major findings, both about the tests, both now addressed in
Two Minor findings deferred, with reasoning:
What still needs a human with a MacThe iOS tests have never been executed. This host has no Worth a separate issue: CI has no iOS test job at all. Also worth knowingObservations captured on Android before this fix still carry boxes in the old |
|
The review follow-up is in
CI run 34479643831 Physical camera/gallery, Android |
|
Final review follow-up: The iOS crop/tensor bridge tests cover all eight EXIF orientations. The native I also tested an approved combined field.6 candidate on Pixel9a Android16 using Descriptions now distinguish current green native CI, the two failure causes, |
|
/gemini review |
Summary
Detection boxes and MiewID crops were computed against the raw sensor pixel
buffer, while the app displays the EXIF-rotated image. For any rotated photo
those are two different pictures, so the coordinates described a grid the user
never sees.
BitmapFactoryignores the EXIF orientation tag; Fresco appliesit (
setAutoRotateEnabled(true)). Boxes were transposed on screen, and cropswere cut from the wrong region.
imageToTensoralready honoured orientation implicitly, becauseresizeImageusesUIImage.draw(in:).cropImageusedimage.cgImage, theraw buffer. The overlay looked correct while the crop handed to MiewID came
from the wrong part of the photo, degrading match scores with no visible
symptom.
The fix applies the orientation tag exactly once, at decode, so the detector,
the overlay, and the crop all share one upright coordinate frame.
Found during review of #34. Independent of that PR and safe to merge in either
order.
Type of Change
Screenshots / Screen Recordings
No UI code changed. For newly processed rotated photos, the visible effect is
that the detector and crop agree with the displayed image. There is no device
capture for this branch yet; on-device before/after on a portrait photo remains
an acceptance check. Existing observations are not repaired by upgrading.
Checklist
General
Testing
npm test)Gates run locally on Windows/WSL:
compileDebugKotlin,lintDebug, and Androidunit tests. The combined
npm testchains iOS tooling unavailable on those hosts.The Android orientation suite now has 12 tests: all eight orientations, the
undefined no-op, two tensor/crop matchability guards, and a
cropImagebridgetest using a real EXIF-tagged JPEG. The bridge regression was verified by
temporarily unwiring orientation handling and observing its failure. These
tests run under
GraphicsMode.NATIVE, because Robolectric's legacy bitmapshadow does not transform pixels for matrix-backed
Bitmap.createBitmap.macOS CI is verified on
0b0a8ab. All four jobs passed inrun 34487570937.
The existing macOS test job runs
npm test, including 1,012 Jest tests, Androidunit tests, and 53 native iOS simulator tests. This includes two new iOS bridge
tests with real JPEGs for all eight EXIF orientations, checking both
cropImageand
imageToTensoragainst independently specified upright quadrant colors.The earlier red job was an existing debug XCTest host starting React Native
without Metro or a bundled script, not an orientation assertion failure. The
follow-up prevents React startup only in debug native-test hosts. Three tests
verify the normal-launch policy, native-test markers, and actual delegate path.
Normal app launches and release startup still start React Native.
A subsequent PR34 run exposed a separate test-fixture race: direct writes to
the download dictionary could overlap background restoration/persistence and
crash with
-[__NSCFNumber count]: unrecognized selector. All three testinjections now use the module's existing barrier queue. This is a test-only
synchronization change, included in the green run above; production download
behavior is unchanged.
React Native Specific
project.pbxproj) (not applicable: none added)useThemedStylespattern (not applicable: no UI code)FlatList/FlashList(unchanged)Performance & Models
Security
Related Issues
Found while reviewing #34. Relates to the detector-threshold calibration work,
which should wait until this lands: comparing on-device detections against
WhiskerBook's
yolo-elephants-head-v0is not meaningful while the two arereading different pixel grids.
Additional Notes
Why the matchability guards. MiewID embeds the cropped pixels. A regression
here produces no crash and no failing screen, just quietly worse match
candidates. Two Android tests pin the invariant directly: the 440x440 input
tensor and the crop region must be byte-identical whether a photo is stored
upright or stored rotated with an orientation tag.
Combined candidate device evidence. An explicitly approved field.6 candidate
(
41c2dca) combines this runtime fix with #34. On Pixel9a Android16, threeGPS-free versions of the same scene (upright, EXIF6 rotation, EXIF2 mirroring)
were imported through the real gallery flow. Boxes differed by about one pixel,
mean crop-channel differences were about 4.3/255, and the top match was unchanged.
These JPEGs were separately encoded, so this is tolerance-based evidence, not
byte-identical inference. Native bridge tests supply the exact orientation oracle.
Known gaps. Real tagged-file bridge coverage now exists on both platforms.
The Pixel gallery test does not prove which URI representation its picker passed
to the native decoder or cover every provider; dedicated Android
content://coverage remains open. Physical camera capture, iPhone and Android9/BlueStacks
validation remain separate. Native simulator tests do not establish those cases.
Memory.
applyExifOrientationallocates a second full-resolution bitmapduring rotation, so peak memory roughly doubles for the moment both exist. The
source is recycled immediately on the success path. Existing full-resolution
decode and exceptional-path recycling need a separate tested follow-up.
Downsampling at decode belongs in its own PR.
Dependency. Adds
androidx.exifinterface:exifinterface:1.3.7, chosen overthe deprecated
android.media.ExifInterfacebecause it reads from streams,which the
content://gallery path needs.Scope. No detector thresholds, weights, pack format, backend contract, or
JavaScript changed. Android observations captured before this fix may still
carry boxes in the old frame; neither platform recomputes existing crops,
embeddings, or review decisions. Inspect affected records before syncing and
do not silently reprocess or clear field data.