Skip to content

Write generated data files as UTF-8 with LF instead of the locale encoding - #1238

Open
ppcvote wants to merge 1 commit into
CERTCC:mainfrom
ppcvote:fix/generated-files-utf8
Open

Write generated data files as UTF-8 with LF instead of the locale encoding#1238
ppcvote wants to merge 1 commit into
CERTCC:mainfrom
ppcvote:fix/generated-files-utf8

Conversation

@ppcvote

@ppcvote ppcvote commented Aug 31, 2026

Copy link
Copy Markdown

Every generated-artifact writer opened its file in text mode with no encoding=, so it used the host's locale encoding. The descriptions' curly quotes exist in cp950 and cp1252, so those writes succeeded and emitted non-UTF-8 bytes with no warning and exit 0. The CSV writer separately lacked newline="", so to_csv's line endings became \r\r\n.

That matters because the doctools-regenerate hook and run_doctools.yml's own failure message both tell a contributor to regenerate and push data/.

Running the hook on a cp950 host, before and after:

before after
files under data/ modified 185 0
no longer valid UTF-8 34 0
CSVs containing \r\r\n 16 0

Seven one-argument changes across doctools.py, md_gen.py and decision_tables/helpers.py: encoding="utf-8" everywhere, newline="\n" on the text writers so output stays LF, and newline="" plus an explicit lineterminator on the two CSV writers. Nothing under data/ is committed here, so run_doctools.yml regenerates and verifies on its own runner.

The test runs a full generation under -X warn_default_encoding (PEP 597) and fails on any EncodingWarning raised from inside the package. That is what makes it work on your UTF-8 CI: asserting on the generated bytes would pass there whether or not the defect is present. It fails on the current code naming doctools.py:345 and :197, and passes with the change.

black --check is clean. The four failures in test_dp_helpers.py and test_mixins.py are present on an unmodified checkout here too.

…oding

doctools opened every generated artifact in text mode with no encoding, so the
descriptions' curly quotes were written in the host code page. Those code points
exist in cp950 and cp1252, so the write succeeded and emitted non-UTF-8 bytes
with no warning. The CSV writer also lacked newline="", turning to_csv's line
endings into \r\r\n.

Running the pre-commit hook on a cp950 host modified 185 files under data/, 34
of which stopped being valid UTF-8. With this change the same run leaves data/
byte-identical.
@ppcvote
ppcvote requested a review from ahouseholder as a code owner August 31, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

doctools writes generated files using the locale encoding, corrupting 34 of them on non-UTF-8 machines

1 participant