fix: strip leading UTF-8 BOM before JSON content-shape detection

[chuenchen309]

Problem

detect_filetype() decodes a file's text head as plain "utf-8". If the source has a leading UTF-8 byte-order-mark, that BOM survives the decode as a literal U+FEFF character at the start of the text.

_TextFileDifferentiator._is_json requires the first non-whitespace character to be [ or {:

if text_head.lstrip()[0] not in "[{":
    return False

str.lstrip() doesn't strip U+FEFF (it's not whitespace), so this check fails immediately for BOM-prefixed content — _is_json returns False without even attempting to parse it. When this happens for a payload with no .json/.ndjson extension and libmagic guesses a generic text/plain MIME type (which it does for BOM-prefixed content — confirmed BOM changes libmagic's own guess from application/json/application/x-ndjson to text/plain), the file falls all the way through to FileType.TXT instead of FileType.JSON.

Repro

import io, json
from unstructured.file_utils.filetype import detect_filetype

json_bytes = b"\xef\xbb\xbf" + json.dumps([{"example": "data"}]).encode("utf-8")
detect_filetype(file=io.BytesIO(json_bytes), metadata_file_path="filename.pdf")
# -> FileType.TXT  (before this fix)
# -> FileType.JSON (after this fix, matching the non-BOM case)

Fix

Strip a leading BOM once in _FileTypeDetectionContext.text_head (the property consumed by _is_json/_is_csv/_is_eml), rather than changing the .encoding property itself — an existing test (it_knows_the_encoding_asserted_by_the_caller_and_normalizes_it) asserts .encoding returns the literal string "utf-8", so I kept that contract untouched and fixed the actual text consumption point instead.

Per the contribution checklist, bumped unstructured/__version__.py to 0.25.1 and added a CHANGELOG.md entry.

Testing

• Added test_it_identifies_json_with_a_leading_utf8_bom_and_no_extension to test_unstructured/file_utils/test_filetype.py; confirmed it fails against the pre-fix code (FileType.TXT instead of FileType.JSON, verified via git stash) and passes with the fix.
• Ran the full test_unstructured/file_utils/ suite (380 passed, 1 xfailed — no regressions) plus test_json.py/test_ndjson.py/test_csv.py/test_text.py (189 passed) since those partitioners consume text_head indirectly through detection.
• ruff check / ruff format --check clean on all changed files; scripts/version-sync.sh -c confirms the version bump is consistent.

Disclosure: I used an AI coding assistant (Claude) to help identify this bug and draft the fix. The initial hypothesis about the exact failure mechanism was actually wrong (it pointed at a different function than the true cause), so I traced through _FileTypeDetector's multi-strategy dispatch myself to find where the fix needed to land, wrote/verified the regression test in both red and green states, and ran the broader test suites above before opening this PR.

[cubic-dev-ai]

All reported issues were addressed across 4 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would auto-approve. Corrects UTF-8 BOM handling in file type detection with a targeted, low-risk change (strip one character) and adds a test. No core logic, infrastructure, or security impact.

Re-trigger cubic</sub>