Skip to content

Fix invalid JSON dump when a string holds non UTF-8 bytes - #141

Open
ousamabenyounes wants to merge 1 commit into
BitOne:masterfrom
ousamabenyounes:fix/issue-124
Open

Fix invalid JSON dump when a string holds non UTF-8 bytes#141
ousamabenyounes wants to merge 1 commit into
BitOne:masterfrom
ousamabenyounes:fix/issue-124

Conversation

@ousamabenyounes

Copy link
Copy Markdown

Fix #124

Problem

meminfo_escape_for_json() escaped only the backslash, the double quote and the
control characters 0x000x1f. It never validated UTF-8.

PHP strings are byte arrays with no encoding guarantee, so an array key, a property
name, a class name, a symbol name or a frame label can hold arbitrary binary data —
a latin-1 accented byte, a truncated multi-byte sequence, a mb_convert_encoding
leftover. Those bytes were written straight into the dump, and because JSON strings
must be valid Unicode the entire dump then failed to decode:

json_last_error_msg(): Malformed UTF-8 characters, possibly incorrectly encoded

One bad byte anywhere in the heap makes the whole dump unusable for every analyzer
command. The two workarounds reported on the issue (iconv -c, or
mb_convert_encoding($json, "UTF-8", "UTF-8")) both repair the file after the fact
and both lose data; neither is reachable from configuration.

Fix

meminfo_escape_for_json() now decodes the input with php_next_utf8_char() and
builds the result in a single pass:

  • a byte that is not part of a valid UTF-8 sequence becomes \ufffd
    (U+FFFD REPLACEMENT CHARACTER) — the same substitution PHP core applies for
    JSON_INVALID_UTF8_SUBSTITUTE;
  • " and \ are backslash-escaped, and 0x000x1f become \u00xx, exactly as
    before;
  • a valid UTF-8 sequence is copied through byte-for-byte, so nothing changes for
    dumps that were already decodable.

Verified equivalence: for lone continuation bytes, 0xC0/0xC1/0xF5/0xFF,
truncated 2/3/4-byte sequences at end of string, overlong encodings, surrogates
(U+D800) and out-of-range code points, the output is byte-for-byte identical to
json_encode($s, JSON_INVALID_UTF8_SUBSTITUTE).

The rewrite also removes the 34 sequential whole-string php_str_to_str() passes the
old implementation made per escaped name (one for \, one for ", then one per
control character). This function runs once per array key, property name and class
name of a dump, so that is ~34 string allocations per name replaced by one.

Test verification (RED → GREEN)

New test: extension/tests/bug-github-124_invalid_utf8_in_array_key.phpt. It asserts
three things — that the dump decodes, that the invalid byte was substituted, and that
a valid multi-byte character was left intact (so a naive "strip everything non-ASCII"
fix would not pass it).

RED — the new test alone, applied to unmodified master, no production change:

TEST 1/1 [tests/bug-github-124_invalid_utf8_in_array_key.phpt]
FAIL Check that bytes that are not valid UTF-8 don't break the JSON dump

---- ACTUAL OUTPUT
meminfo_dump JSON decode fail: Malformed UTF-8 characters, possibly incorrectly encoded
---- EXPECTED OUTPUT
meminfo_dump JSON decode ok
invalid byte replaced: yes
valid utf-8 preserved: yes

Tests failed    :    1 (100.0%)
Tests passed    :    0 (  0.0%)

GREEN — same test on this branch:

TEST 1/1 [tests/bug-github-124_invalid_utf8_in_array_key.phpt]
PASS Check that bytes that are not valid UTF-8 don't break the JSON dump

Tests skipped   :    0 (  0.0%)
Tests failed    :    0 (  0.0%)
Tests passed    :    1 (100.0%)

The test also fails again if the production change alone is reverted while keeping the
test, so it is genuinely bound to the fix.

Full suite, on the CI matrix

phpize && ./configure --enable-meminfo && make && make test, plus
cd analyzer && composer install && vendor/bin/phpspec run, run in Docker on every
PHP version from the workflow matrix:

PHP master this branch
7.0 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped
7.1 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped
7.2 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped
7.3 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped
7.4 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped
8.0 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped
8.1 13 passed / 0 failed / 0 skipped 14 passed / 0 failed / 0 skipped

Analyzer phpspec: 6 examples / 6 passed, before and after. No compiler warnings on any
of the seven versions. The three pre-existing tests that already cover this function
(dump-check_json_escape.phpt, dump-check_json_unicode.phpt,
bug-github-49_escape_control_character.phpt) still pass unchanged.

Dumping an identical script through a master build and a patched build and diffing
every emitted JSON string produces no difference, so valid input is untouched: / is
still not escaped, 0x7f is still raw, and 2/3/4-byte UTF-8 class and property names
are unchanged.

A gcov build reports 100% line coverage of the added code by the test suite, with each
of the four branches (invalid sequence, quote/backslash, control character, valid
passthrough) actually taken.

Notes

  • U+FFFD loses the original byte value. The alternative — emitting the byte as
    \u00XX — would silently assert the data was latin-1, which is a guess; following
    what PHP core itself does for invalid UTF-8 seemed the safer default. Happy to
    switch if you would rather keep the raw byte value.
  • Deliberately out of scope: a key consisting only of a NUL byte still collapses to an
    empty key, because the function takes a const char * and uses strlen(). That
    behaviour is identical before and after this change, and fixing it means changing
    the signature to carry a length, which is a larger change than this issue needs.

meminfo_escape_for_json() escaped only the backslash, the double quote and
control characters, so any byte that is not valid UTF-8 in an array key,
property name, class name, symbol name or frame label was written raw into the
dump and json_decode() failed with "Malformed UTF-8 characters, possibly
incorrectly encoded" (GitHub BitOne#124). Such bytes are now substituted with U+FFFD,
the same substitution PHP core applies for JSON_INVALID_UTF8_SUBSTITUTE.
@DocDocTeam

DocDocTeam commented Sep 7, 2026 via email

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Malformed UTF-8 characters, possibly incorrectly encoded

2 participants