Fix invalid JSON dump when a string holds non UTF-8 bytes - #141
Open
ousamabenyounes wants to merge 1 commit into
Open
Fix invalid JSON dump when a string holds non UTF-8 bytes#141ousamabenyounes wants to merge 1 commit into
ousamabenyounes wants to merge 1 commit into
Conversation
meminfo_escape_for_json() escaped only the backslash, the double quote and control characters, so any byte that is not valid UTF-8 in an array key, property name, class name, symbol name or frame label was written raw into the dump and json_decode() failed with "Malformed UTF-8 characters, possibly incorrectly encoded" (GitHub BitOne#124). Such bytes are now substituted with U+FFFD, the same substitution PHP core applies for JSON_INVALID_UTF8_SUBSTITUTE.
|
Добрый день.
Отсутствую 07.09.26. На все ваши вопросы обязательно отвечу 08.09.26.
По очень срочным вопросам прошу обращаться к Анне Латышевой и Татьяне Кубаревой - ***@***.***
Если вопрос не очень срочный, прошу подождать меня.
Спасибо.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #124
Problem
meminfo_escape_for_json()escaped only the backslash, the double quote and thecontrol characters
0x00–0x1f. It never validated UTF-8.PHP strings are byte arrays with no encoding guarantee, so an array key, a property
name, a class name, a symbol name or a frame label can hold arbitrary binary data —
a latin-1 accented byte, a truncated multi-byte sequence, a
mb_convert_encodingleftover. Those bytes were written straight into the dump, and because JSON strings
must be valid Unicode the entire dump then failed to decode:
One bad byte anywhere in the heap makes the whole dump unusable for every analyzer
command. The two workarounds reported on the issue (
iconv -c, ormb_convert_encoding($json, "UTF-8", "UTF-8")) both repair the file after the factand both lose data; neither is reachable from configuration.
Fix
meminfo_escape_for_json()now decodes the input withphp_next_utf8_char()andbuilds the result in a single pass:
\ufffd(U+FFFD REPLACEMENT CHARACTER) — the same substitution PHP core applies for
JSON_INVALID_UTF8_SUBSTITUTE;"and\are backslash-escaped, and0x00–0x1fbecome\u00xx, exactly asbefore;
dumps that were already decodable.
Verified equivalence: for lone continuation bytes,
0xC0/0xC1/0xF5/0xFF,truncated 2/3/4-byte sequences at end of string, overlong encodings, surrogates
(
U+D800) and out-of-range code points, the output is byte-for-byte identical tojson_encode($s, JSON_INVALID_UTF8_SUBSTITUTE).The rewrite also removes the 34 sequential whole-string
php_str_to_str()passes theold implementation made per escaped name (one for
\, one for", then one percontrol character). This function runs once per array key, property name and class
name of a dump, so that is ~34 string allocations per name replaced by one.
Test verification (RED → GREEN)
New test:
extension/tests/bug-github-124_invalid_utf8_in_array_key.phpt. It assertsthree things — that the dump decodes, that the invalid byte was substituted, and that
a valid multi-byte character was left intact (so a naive "strip everything non-ASCII"
fix would not pass it).
RED — the new test alone, applied to unmodified
master, no production change:GREEN — same test on this branch:
The test also fails again if the production change alone is reverted while keeping the
test, so it is genuinely bound to the fix.
Full suite, on the CI matrix
phpize && ./configure --enable-meminfo && make && make test, pluscd analyzer && composer install && vendor/bin/phpspec run, run in Docker on everyPHP version from the workflow matrix:
masterAnalyzer phpspec: 6 examples / 6 passed, before and after. No compiler warnings on any
of the seven versions. The three pre-existing tests that already cover this function
(
dump-check_json_escape.phpt,dump-check_json_unicode.phpt,bug-github-49_escape_control_character.phpt) still pass unchanged.Dumping an identical script through a
masterbuild and a patched build and diffingevery emitted JSON string produces no difference, so valid input is untouched:
/isstill not escaped,
0x7fis still raw, and 2/3/4-byte UTF-8 class and property namesare unchanged.
A gcov build reports 100% line coverage of the added code by the test suite, with each
of the four branches (invalid sequence, quote/backslash, control character, valid
passthrough) actually taken.
Notes
U+FFFDloses the original byte value. The alternative — emitting the byte as\u00XX— would silently assert the data was latin-1, which is a guess; followingwhat PHP core itself does for invalid UTF-8 seemed the safer default. Happy to
switch if you would rather keep the raw byte value.
empty key, because the function takes a
const char *and usesstrlen(). Thatbehaviour is identical before and after this change, and fixing it means changing
the signature to carry a length, which is a larger change than this issue needs.