Skip to content

ci: parse LuaLS json output instead of the pretty output - #7877

Open
Rathoz wants to merge 6 commits into
mainfrom
chore/fix-luals-problem-count
Open

ci: parse LuaLS json output instead of the pretty output#7877
Rathoz wants to merge 6 commits into
mainfrom
chore/fix-luals-problem-count

Conversation

@Rathoz

@Rathoz Rathoz commented Jul 29, 2026

Copy link
Copy Markdown
Member

What

The luals-code-style step scraped LuaLS's human-readable output — stripping ANSI codes, then pulling counts out of prose. Now it uses --check_format=json and jq.

  • Counts come from the JSON, so they're exact. The old grep -c 'Warning' reported 263 where LuaLS itself said 238 (one diagnostic can span several lines).
  • Annotations and the pass/fail decision are now explicit. Previously the ::error lines were piped into grep, so they only reached the log because grep echoes what it matches, and the exit status came from inverting that grep.
  • Summary gains a count of problems in changed files — the number that actually gates.
Files scanned Files with problems Problems In changed files
LuaLS check 1710 142 238 0

Gating behaviour is unchanged: annotate and fail only for files the PR touches.

How it was tested

luals-code-style skips unless lua/wikis/**/*.lua changes, so both paths were verified on a throwaway branch (since deleted):

  • Clean changed fileIn changed files: 0, job passes
  • Changed file with a known problemIn changed files: 1, job fails, and GitHub registered the annotation at FilterButtons/Config.lua:27 — the exact line predicted locally

Also checked locally: JSON line/character are 0-based where annotations are 1-based (verified both 27:23 and 42:2 against the pretty output), a clean run writes [] rather than {} so the count needs // 0, and multi-line messages are flattened so they don't break the workflow command.

The step summary counted lines containing "Warning":

    $(cat luals-check | grep 'Warning' | wc -l)

That gives 263 where LuaLS itself reports 238, because a single diagnostic can
span several lines. Use the totals LuaLS prints, and add files-with-problems
alongside them.

No delta is reported against the base, because the count is not deterministic:
237-240 across 8 runs on identical code, in both pretty and json output, with no
--threads flag to pin the analysis. Files-with-problems was stable at 142 in
every run, so that is the number worth watching, and the summary says as much.

Gating behaviour is untouched; only the summary lines change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 29, 2026 09:15
@Rathoz
Rathoz requested review from a team as code owners July 29, 2026 09:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Rathoz added a commit that referenced this pull request Jul 29, 2026
Two unrelated changes were sharing a branch: a new metric and a fix to an
existing one. Split so they can be reviewed and reverted independently.

The LuaLS summary fix now lives in #7877. luals-code-style here is byte
identical to main again; this branch only adds annotation coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rathoz
Rathoz requested a review from mbergen July 29, 2026 13:01
@mbergen

mbergen commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

I didn't know there is a JSON format available for output (it's not in the documentation on luals.github.io).
Using that and some jq parsing would probably improve the codestyle of this action drastically, so preferably we switch to that.

Per review (@mbergen): --check_format=json plus jq, rather than stripping ANSI
codes and pulling totals out of prose.

This removes the reason for the previous commit. Counting lines containing
"Warning" gave 263 against LuaLS's own 238, because one diagnostic can span
several lines; counting array elements cannot go wrong that way, so the fix
becomes moot rather than needing to be gotten right.

The old pipeline also only surfaced annotations as a side effect: the ::error
lines were piped into grep, so they reached the log only because grep echoes
the lines it matches, and the exit status came from inverting that grep. Now
annotations and the pass/fail decision are both explicit, and the summary gains
a count of problems in changed files -- the number that actually gates.

Three details that needed checking rather than assuming:

- JSON line/character are 0-based where the pretty output and the annotations
  are 1-based, so both need +1. Verified against pretty for two files:
  27:23 and 42:2 either way.
- A clean run writes `[]`, not `{}`, so `[.[] | length] | add` yields null and
  the summary would say "null problems". Hence `// 0`.
- Messages can contain newlines, which would break the ::error command, so
  they are flattened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rathoz Rathoz changed the title ci: report the LuaLS problem count LuaLS actually reports ci: parse LuaLS json output instead of the pretty output Sep 7, 2026
@Rathoz

Rathoz commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

I didn't know there is a JSON format available for output (it's not in the documentation on luals.github.io). Using that and some jq parsing would probably improve the codestyle of this action drastically, so preferably we switch to that.

Done in 3f92ac7 — good call, it's a much better fit.

It also removes the reason this PR existed. The original fix was to stop grep -c 'Warning' overcounting (263 vs 238). Counting JSON array elements can't go wrong that way, so the bug is gone rather than corrected.

One thing worth knowing about the old pipeline while we're here:

sed -n 's/.../::error file=,line=/p' | (! grep -E "$changed_files")

The ::error lines were piped into grep, so they reached the log only because grep echoes the lines it matched, and the exit status came from inverting that grep. The behaviour was right, but by side effect. Both are now explicit, and the summary gained an In changed files column — the number that actually decides pass/fail.

Three details that needed checking rather than assuming:

  • JSON line/character are 0-based; annotations are 1-based. Verified both ways against the pretty output (27:23, 42:2).
  • A clean run writes [], not {}, so [.[] | length] | add yields null and the summary would have said "null problems". Needs // 0.
  • Messages can contain newlines, which break the ::error command, so they're flattened.

Verified in CI on a throwaway branch, both paths: a clean changed file passes with In changed files: 0; a changed file with a known problem fails, and GitHub registered the annotation at FilterButtons/Config.lua:27.

Also — you're right that it isn't on luals.github.io. It's only in --help, alongside --check_out_path.

Comment thread .github/workflows/luacheck.yml Outdated
Rathoz and others added 2 commits September 7, 2026 13:00
Removing one echo left the sentence hanging mid-clause with no closing
underscore, so the summary rendered a broken italic. Taking the other half out
as well.

The caveat itself still holds -- the count moved between 237 and 240 across 8
runs on identical code -- it just does not need to be restated on every run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants