exposure fusion - xiao esp32s3 sense

image signal processing project implementing mertens exposure fusion directly on the seeed studio xiao esp32s3 sense microcontroller with its onboard ov2640 camera.

what it does

the board captures three frames of the same scene at different exposure settings: dark, balanced, and bright. a pixel-wise weighted blend merges all three into a single well-exposed image. the result is served over wi-fi as a downloadable bmp file so you can inspect it from any phone or laptop without any usb connection.

this is a fully embedded implementation with no pc-side processing. everything runs on the esp32-s3.

hardware setup steps

here is the board assembly and antenna connection:

pc output cam footage

here is the live camera feed and object detection output:

algorithm: mertens exposure fusion

the core idea comes from mertens et al. (2007): assign each pixel from each exposure a well-exposedness weight based on how close its luminance is to the midpoint (0.5 on a 0-1 scale). pixels that are neither clipped white nor crushed black score highest.

weight(R, G, B) = exp( -(L - 0.5)^2 / (2 * sigma^2) ) where sigma = 0.2, L = mean(R,G,B)/255

for each output pixel: fused_channel = (C1w1 + C2w2 + C3*w3) / (w1 + w2 + w3)

this recovers detail from both shadows and highlights in a single image. you can find other algorithms to study in the algorithm/ folder.

project structure

fusion_algorithm/
├── platformio.ini              build config for seeed_xiao_esp32s3
├── src/
│   ├── main.cpp                exposure fusion firmware (main entry point)
│   ├── ExposureFusion.cpp      algorithm implementation
│   └── ExposureFusion.h        algorithm header
├── algorithm/                  study files for other isp algorithms
│   ├── simple_average.cpp
│   ├── hdr_radiance.cpp
│   └── tone_mapping.cpp
└── tools/
    ├── simulate_fusion.py      python script for testing fusion logic
    └── download_fused.ps1      script to grab result via wi-fi

src/main.cpp is the one file you care about for the firmware.

how to flash

1. install platformio in vs code.
2. connect xiao esp32-s3 sense via usb-c.
3. open this folder in vs code.
4. click upload or run pio run -t upload in the terminal.
5. open the serial monitor at 115200 baud.

using the result

1. join the xiao_fusion wi-fi network from your phone or laptop (password 12345678).
2. open http://192.168.4.1/fused.bmp in your browser.

key design decisions

why rgb565 internally? the ov2640 natively outputs rgb565. staying in that format avoids a conversion step and keeps memory usage at 2 bytes per pixel.

why bmp output? bmp has no decoder dependency. every os and browser opens it without plugins.

why psram? four 160x120x2-byte buffers fits comfortably in the 8 mb psram on the sense variant but would exceed internal dram.

why soft-ap instead of sta mode? no router configuration required. you point your phone directly at the board.

limitations and next steps

• resolution is capped at qqvga (160x120) to keep processing fast and ram usage low. qvga (320x240) is possible with psram.
• the three exposures are sequential captures, so any scene motion between shots will cause ghosting. a multi-frame anti-ghosting pass could be added.
• aec convergence is handled with simple delay calls. reading the sensor register to confirm lock would be more robust.