Log worker records as JSON with a pluggable formatter - #44
Merged
Conversation
Replace the hardcoded multiprocessing log formatter with a new JsonFormatter default and a TextFormatter preset. The formatter is swappable via TaskExecutor(log_formatter=...) and the new --log-format json|text worker option; spawned worker processes receive and apply the configured formatter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the json|text presets with a free-form logging format string: --log-format '%(levelname)s %(message)s' wraps the string in logging.Formatter, while omitting the option keeps the JSON default. Invalid format strings fail fast with CommandError via a smoke-test emit on a real LogRecord, before any handler is touched. TaskExecutor.log_formatter remains the single owner of the JSON default; WorkerProcess now requires the formatter as a keyword-only argument and applies it directly on the shared handler. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
codingjoe
commented
Sep 7, 2026
Serialize log records with DjangoJSONEncoder instead of casting the datetime to an ISO string, and stamp records with Django's current timezone rather than hardcoded UTC. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Merge non-standard LogRecord attributes (set via logging's extra={})
into the JSON payload as top-level fields. Values must be
JSON-serializable, matching how normalize_json treats task return
values.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The multiprocessing logger's format was hardcoded to a single text layout. This PR makes worker logs JSON by default (one object per line, easy to ingest for log collectors) while keeping the formatter pluggable, as requested.
What changed
JsonFormatter(new default): one JSON object per log record withcreated_at(ISO 8601 UTC),level,logger,message,process,process_name,thread, plus anexceptionfield with the traceback when present.TaskExecutor(log_formatter=...)accepts anylogging.Formatterinstance and is the single owner of the JSON default. The formatter is passed to everyWorkerProcessand applied in both the parent and child processes, so workers spawned via the spawn start method (which re-import the module) log with the same formatter.manage.py threadmill worker --log-format '%(levelname)s %(message)s'accepts a free-form logging format string for human-readable output. Omitting the option keeps JSON.Non-obvious details
CommandError.logging.Formatteronly checks that the string contains at least one valid field, so mixed strings like'%(message)s 100% done'pass construction but raiseTypeErroron every emit, silently dropping all log records in the parent and all workers. The command therefore smoke-tests the formatter on a realLogRecordand catchesTypeError/ValueErrorbefore any handler is touched.--log-format ''is honored as a format string (message-only output); only the absent option falls back to JSON.WorkerProcessnow requireslog_formatteras a keyword-only argument.logging.Formatterinstances are picklable, which is what makes the spawn path work.Tests cover every branch of the new option (format string, JSON default, empty string, invalid input) and keep patch coverage at 100%.