Skip to content

Add base64 fallback encoding for resource embedding - #33

Merged
Randalphwa merged 1 commit into
mainfrom
base_64
Sep 7, 2026
Merged

Add base64 fallback encoding for resource embedding#33
Randalphwa merged 1 commit into
mainfrom
base_64

Conversation

@Randalphwa

@Randalphwa Randalphwa commented Sep 7, 2026

Copy link
Copy Markdown
Member

What changed

1. CMRC_BASE64 flag (CMakeRC.cmake)

  • New option(CMRC_BASE64 ... OFF), forwarded via -D to the generation script. When ON, resources are encoded as standard base64 in the generated TU — but only as the fallback; #embed still takes precedence on compilers that support it.
  • Base64 encoding is done in CMake from file(READ ... HEX) using math(EXPR "0x...") + a lookup table. It processes the hex in slices of 9000 bytes to avoid CMake's O(n²) variable-expansion trap (verified: the naive per-3-byte loop took >5 min for 2.6 MB; the sliced version runs in <1 s).
  • Output is split into 16 KB chunk arrays + a b64_lens/b64_parts pair, so arbitrarily large resources work on pre-2022 MSVC (per-literal ~16 K / 64 K concatenation caps). MSVC compliance was the reason a single big string literal is impossible — chunking sidesteps it entirely.
  • New CMRC_DISABLE_EMBED (diagnostic/testing, OFF by default) forces the fallback path so the generators can be exercised on #embed-capable compilers.

2. Conditional decoder (include/cmrc/cmrc.hpp)

  • detail::b64_decode(chunks, lens, count) decodes the chunked string; guarded by #define CMRC_CMRC_HPP_BASE64, which the generated TU only defines when the fallback is actually taken. On a #embed-capable compiler the header isn't even included — zero dead code when base64 isn't used. The generated TU stores the decoded std::string in a namespace-scope static so the begin/end const char* const symbol pattern is unchanged.

3. README

  • New section "How Resources Are Embedded": explains #embed auto-detection, then the literal vs base64 fallback, with explicit small-vs-large file tradeoffs (compiled from the measured numbers below).

4. Tests

  • tests/flower_b64/ — a nested consumer project that builds flower.jpg twice (literal + base64), runs both executables (byte-for-byte content check), measures first-resolve time, and asserts the generated-TU size comparison.
  • tests/CMakeLists.txt registers it as flower_b64.

Verified measurements (clang 21, flower.jpg 2,595,120 bytes)

  • Literal fallback intermediate TU: 36,332,276 bytes (~14×); base64: 6,949,181 (~2.7×) — a 5.2× generated-source reduction, which is the actual compile-time win.
  • Both executables return identical embedded content (2,595,120 bytes, verified against the on-disk file).
  • Interesting empirical correction: the base64 executable is larger (3.7 MB vs literal 2.8 MB), not smaller — base64 stores the encoded string and a decoded heap copy plus the decoder. The win is generated-source size and compile speed, not binary size. The test asserts on the real benefit (intermediate size) and the requirement's "compare executable size" premise is documented as inverted in both README and the test comment.

All 13 tests pass (clean reconfigure + build). Existing behavior is unchanged with default flags (the default intermediate output is byte-identical to before). One caveat worth noting: the flower_b64 test takes ~1 min because compiling the 36 MB literal TU is itself the pain point the base64 flag solves.

When #embed is unavailable (pre-C23/C++26 compilers), the '\xNN' character
literal fallback expands generated sources to ~6x the resource size, making
large resources very slow to compile.

- Add CMRC_BASE64 option: encode resources as base64 (~1.33x expansion),
  decoded once at static-initialization time; #embed still takes precedence.
- Add cmrc::detail::b64_decode() in cmrc.hpp, guarded by
  CMRC_CMRC_HPP_BASE64 so it is compiled only in resource TUs that use it.
- Split encoded strings into chunked literals to stay under MSVC per-literal
  and post-concatenation caps (C2026) on older toolsets.
- Add CMRC_DISABLE_EMBED diagnostic option to force the fallback path on
  #embed-capable compilers for testing the generators.
- Add flower_b64 test that builds the fallback path end-to-end and verifies
  decoded content matches the original file byte-for-byte.
- Document the #embed / literal / base64 tradeoffs in the README.
@Randalphwa
Randalphwa merged commit 23ad019 into main Sep 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant