Skip to content

fix: put detector, overlay, and crop on one EXIF-upright grid - #35

Merged
JasonWildMe merged 5 commits into
wildlife-reidfrom
fix/exif-orientation-coordinate-frame
Sep 10, 2026
Merged

fix: put detector, overlay, and crop on one EXIF-upright grid#35
JasonWildMe merged 5 commits into
wildlife-reidfrom
fix/exif-orientation-coordinate-frame

Conversation

@JasonWildMe

@JasonWildMe JasonWildMe commented Sep 10, 2026

Copy link
Copy Markdown

Summary

Detection boxes and MiewID crops were computed against the raw sensor pixel
buffer, while the app displays the EXIF-rotated image. For any rotated photo
those are two different pictures, so the coordinates described a grid the user
never sees.

  • Android: BitmapFactory ignores the EXIF orientation tag; Fresco applies
    it (setAutoRotateEnabled(true)). Boxes were transposed on screen, and crops
    were cut from the wrong region.
  • iOS: imageToTensor already honoured orientation implicitly, because
    resizeImage uses UIImage.draw(in:). cropImage used image.cgImage, the
    raw buffer. The overlay looked correct while the crop handed to MiewID came
    from the wrong part of the photo, degrading match scores with no visible
    symptom.

The fix applies the orientation tag exactly once, at decode, so the detector,
the overlay, and the crop all share one upright coordinate frame.

Found during review of #34. Independent of that PR and safe to merge in either
order.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (code change that neither fixes a bug nor adds a feature)
  • Chore (build process, CI, dependency updates, etc.)

Screenshots / Screen Recordings

No UI code changed. For newly processed rotated photos, the visible effect is
that the detector and crop agree with the displayed image. There is no device
capture for this branch yet; on-device before/after on a portrait photo remains
an acceptance check. Existing observations are not repaired by upgrading.

Checklist

General

  • My code follows the project's coding style and conventions
  • I have performed a self-review of my code
  • I have added/updated comments where the logic isn't self-evident
  • My changes generate no new warnings or errors

Testing

  • I have tested on Android (physical device or emulator)
  • I have tested on iOS (native simulator tests in CI; not UI or physical-device acceptance)
  • I have tested in light mode and dark mode
  • Existing tests pass locally (npm test)
  • I have added tests that prove my fix is effective or my feature works

Gates run locally on Windows/WSL: compileDebugKotlin, lintDebug, and Android
unit tests. The combined npm test chains iOS tooling unavailable on those hosts.

The Android orientation suite now has 12 tests: all eight orientations, the
undefined no-op, two tensor/crop matchability guards, and a cropImage bridge
test using a real EXIF-tagged JPEG. The bridge regression was verified by
temporarily unwiring orientation handling and observing its failure. These
tests run under GraphicsMode.NATIVE, because Robolectric's legacy bitmap
shadow does not transform pixels for matrix-backed Bitmap.createBitmap.

macOS CI is verified on 0b0a8ab. All four jobs passed in
run 34487570937.
The existing macOS test job runs npm test, including 1,012 Jest tests, Android
unit tests, and 53 native iOS simulator tests. This includes two new iOS bridge
tests with real JPEGs for all eight EXIF orientations, checking both cropImage
and imageToTensor against independently specified upright quadrant colors.

The earlier red job was an existing debug XCTest host starting React Native
without Metro or a bundled script, not an orientation assertion failure. The
follow-up prevents React startup only in debug native-test hosts. Three tests
verify the normal-launch policy, native-test markers, and actual delegate path.
Normal app launches and release startup still start React Native.

A subsequent PR34 run exposed a separate test-fixture race: direct writes to
the download dictionary could overlap background restoration/persistence and
crash with -[__NSCFNumber count]: unrecognized selector. All three test
injections now use the module's existing barrier queue. This is a test-only
synchronization change, included in the green run above; production download
behavior is unchanged.

React Native Specific

  • No new native module without corresponding platform implementation (Android + iOS)
  • New native modules are added to the Xcode project build target (project.pbxproj) (not applicable: none added)
  • No hardcoded pixel values (not applicable: no UI code)
  • Styles use useThemedStyles pattern (not applicable: no UI code)
  • Animations/gestures work smoothly on both platforms (not applicable)
  • Large lists use FlatList / FlashList (unchanged)
  • No unnecessary re-renders introduced (no JS changed)

Performance & Models

  • Downloads / long-running tasks report progress to the UI (unchanged)
  • File paths are resolved correctly on both platforms (unchanged)
  • Large files (models, assets) are not committed to the repository

Security

  • No secrets, API keys, or credentials are included in the code
  • User input is validated/sanitized where applicable

Related Issues

Found while reviewing #34. Relates to the detector-threshold calibration work,
which should wait until this lands: comparing on-device detections against
WhiskerBook's yolo-elephants-head-v0 is not meaningful while the two are
reading different pixel grids.

Additional Notes

Why the matchability guards. MiewID embeds the cropped pixels. A regression
here produces no crash and no failing screen, just quietly worse match
candidates. Two Android tests pin the invariant directly: the 440x440 input
tensor and the crop region must be byte-identical whether a photo is stored
upright or stored rotated with an orientation tag.

Combined candidate device evidence. An explicitly approved field.6 candidate
(41c2dca) combines this runtime fix with #34. On Pixel9a Android16, three
GPS-free versions of the same scene (upright, EXIF6 rotation, EXIF2 mirroring)
were imported through the real gallery flow. Boxes differed by about one pixel,
mean crop-channel differences were about 4.3/255, and the top match was unchanged.
These JPEGs were separately encoded, so this is tolerance-based evidence, not
byte-identical inference. Native bridge tests supply the exact orientation oracle.

Known gaps. Real tagged-file bridge coverage now exists on both platforms.
The Pixel gallery test does not prove which URI representation its picker passed
to the native decoder or cover every provider; dedicated Android content://
coverage remains open. Physical camera capture, iPhone and Android9/BlueStacks
validation remain separate. Native simulator tests do not establish those cases.

Memory. applyExifOrientation allocates a second full-resolution bitmap
during rotation, so peak memory roughly doubles for the moment both exist. The
source is recycled immediately on the success path. Existing full-resolution
decode and exceptional-path recycling need a separate tested follow-up.
Downsampling at decode belongs in its own PR.

Dependency. Adds androidx.exifinterface:exifinterface:1.3.7, chosen over
the deprecated android.media.ExifInterface because it reads from streams,
which the content:// gallery path needs.

Scope. No detector thresholds, weights, pack format, backend contract, or
JavaScript changed. Android observations captured before this fix may still
carry boxes in the old frame; neither platform recomputes existing crops,
embeddings, or review decisions. Inspect affected records before syncing and
do not silently reprocess or clear field data.

BitmapFactory and UIImage.cgImage return the stored sensor buffer and
ignore the EXIF orientation tag, while React Native's <Image> and every
EXIF-aware viewer apply it. Detection boxes were normalized against one
grid and drawn on another: transposed overlays on Android, and on iOS a
correct-looking overlay over crops cut from the wrong region, so MiewID
embedded the wrong pixels and match scores degraded silently.

Android: apply the orientation tag once in loadBitmap, so imageToTensor
and cropImage both work from the display grid.

iOS: normalize orientation before cropping, matching what imageToTensor
already did implicitly through resizeImage.

Adds 11 Robolectric tests covering all eight EXIF orientations plus two
matchability guards asserting that the MiewID input tensor and the crop
region are identical whether a photo is stored upright or rotated.
Robolectric's legacy bitmap shadow does not transform pixels for
matrix-backed createBitmap, so these run under GraphicsMode.NATIVE in
their own class.

The iOS tests are written but unverified: this host has no xcodebuild
and CI has no iOS test job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JasonWildMe and others added 2 commits September 9, 2026 23:02
Both gaps came out of a Codex review of the fix.

The orientation tests all called applyExifOrientation and uprightImage
directly, so removing the call from loadBitmap or cropImage would have
left every one of them green. Adds an end-to-end test through the
cropImage bridge method using a real EXIF-tagged JPEG. Verified by
unwiring loadBitmap: exactly that test fails, and it passes again once
restored. It goes through cropImage rather than imageToTensor because
imageToTensor resolves a WritableNativeArray, which needs React
Native's JNI that Robolectric does not load.

The iOS fixtures used UIGraphicsImageRenderer's default screen scale
while rewrapping the buffer at scale 1, so a "4x2" image is an 8x4 or
12x6 buffer on a 2x or 3x simulator and the dimension assertions would
fail there despite correct production code. Both fixtures now render at
an explicit scale of 1.

Android: 12 orientation tests pass, whole suite green. The iOS tests
remain unverified on this host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the verdict, the two Major test gaps and how each was closed,
the two Minor items deferred with reasoning, and the open gaps -- chiefly
that the iOS tests have never run, because this host has no xcodebuild
and CI has no iOS test job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@JasonWildMe

Copy link
Copy Markdown
Author

Codex second-opinion review

Ran a scoped Codex review over the Android transform, the iOS normalization, the
tests, and the pipeline coordinate contract. Verdict: "The orientation fix is
correct and converged geometrically. The tests are not fully converged."
No
Critical defect. Full report is in
kb/wildlife-reid-mobile/outputs/reports/2026-09-09-codex-exif-orientation-review.md.

It independently derived the source-to-upright pixel mapping for all eight EXIF
orientations and confirmed every stored fixture is the correct inverse of its
upright fixture, including the post-multiplication order that makes
Fx·R90 a transpose and Fx·R270 a transverse.

Two Major findings, both about the tests, both now addressed in 7fa68d2:

  1. Every test called applyExifOrientation / uprightImage directly, so
    deleting the call from loadBitmap or cropImage would have left them all
    green. Added an end-to-end test through the cropImage bridge method with a
    real EXIF-tagged JPEG. I verified it by unwiring loadBitmap: exactly that
    one test fails, and it passes again when restored. It uses cropImage
    rather than imageToTensor because the latter resolves a
    WritableNativeArray, which needs React Native's JNI that Robolectric does
    not load.
  2. The iOS fixtures rendered at the default screen scale while rewrapping the
    buffer at scale: 1, so a "4x2" image is an 8x4 or 12x6 buffer on a 2x or
    3x simulator and the dimension assertions would fail there despite correct
    production code. Both fixtures now render at an explicit scale of 1.

Two Minor findings deferred, with reasoning:

  • Exceptional paths skip bitmap recycling, so a transform or file-write failure
    leaves allocations for the GC rather than releasing them deterministically.
    Successful-path ownership is correct, with no double-recycle or
    use-after-recycle. Making release deterministic is a behaviour change that
    deserves its own tests rather than riding along here.
  • Crop quantization differs across platforms: Android truncates origin and
    extent independently, iOS hands fractional bounds to Core Graphics, which
    expands to integral bounds. For x=1.5, width=2.5 Android takes [1,3) and
    iOS [1,4). Pre-existing and unrelated to EXIF; the fix is a shared
    integer-bound calculation on both platforms.

What still needs a human with a Mac

The iOS tests have never been executed. This host has no xcodebuild and no
Swift toolchain, and the CI workflow has no iOS test job, so nothing has run
them. That is the main thing I cannot close myself. There is also no iOS
call-site test yet, so finding 1 above is closed for Android only.

Worth a separate issue: CI has no iOS test job at all.

Also worth knowing

Observations captured on Android before this fix still carry boxes in the old
coordinate frame. They are not migrated, and nothing records which convention
produced them.

@BMichaelJ

Copy link
Copy Markdown
Collaborator

The review follow-up is in aa4c62a.

  • Added real EXIF-tagged JPEG tests through iOS cropImage and imageToTensor,
    covering all eight orientations with independent expected quadrant colors.
  • Fixed the existing iOS native-test host starting React Native without Metro.
    The guard is debug/XCTest-only; normal and release app startup are unchanged.
  • Corrected the CI notes and documented that existing observations, crops,
    embeddings and review decisions are not migrated.

CI run 34479643831
passed all four jobs on this commit. All 53 native iOS tests passed, including
the two bridge tests and three startup tests. No local Windows Swift pass or
physical-device acceptance is implied.

Physical camera/gallery, Android content://, and Android9/BlueStacks checks
remain open. No merge, new device installation, or distribution performed.

@BMichaelJ

Copy link
Copy Markdown
Collaborator

Final review follow-up: 0b0a8ab is green across all four jobs in
run 34487570937.

The iOS crop/tensor bridge tests cover all eight EXIF orientations. The native
test-host guard fixed the missing-Metro startup issue; a later PR34 run exposed
another failure, a test-only dictionary write racing background restoration.
All three download-fixture injections now use the module's existing barrier
queue. No production download code or your orientation transforms were changed.

I also tested an approved combined field.6 candidate on Pixel9a Android16 using
upright, rotated and mirrored copies of one GPS-free scene. The displayed boxes
and actual crops agreed closely, with the same top match; offline restart and
all four existing records were preserved. Three new TEST ONLY records remain
pending/unsynced. This is real gallery-flow evidence, not a claim of every
content-provider path, physical iPhone, Android9/BlueStacks or camera coverage.

Descriptions now distinguish current green native CI, the two failure causes,
and the fact that old observations/crops/embeddings are not recomputed. No PR
merge, approval, or candidate distribution performed.

@BMichaelJ

Copy link
Copy Markdown
Collaborator

/gemini review

@JasonWildMe
JasonWildMe merged commit 68cc76f into wildlife-reid Sep 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants