validate-agent.sh: don't abort at the first warning (set -e + ((x++))) and stop false-flagging valid agents

[bcherny]

Fixes public issue #83803

The plugin-dev skill's validate-agent.sh failed on plugin-dev's own agent files. Three root causes, all set -euo pipefail interactions:

1. Abort at the first warning. ((warning_count++)) / ((error_count++)) evaluate the arithmetic expression, and ((expr)) returns a nonzero exit status when the expression's value is 0 — so the first increment from 0 killed the script under set -e, mid-run, with exit 1. All increments now use count=$((count + 1)), which is an assignment and always returns 0.

2. Abort on any absent frontmatter field. Extractions like TOOLS=$(echo "$FRONTMATTER" | grep '^tools:' | ...) propagate grep's exit 1 (no match) into the assignment, aborting the script instead of reporting the missing field. Each extraction now ends in || true.

3. False "missing <example> blocks" warning. DESCRIPTION was extracted with grep '^description:', which only captures the first physical line — but plugin-dev's own agents use multi-line descriptions whose <example> blocks sit on later lines, so valid agents were flagged. The extraction now captures the full multi-line value (from description: up to the next top-level agent key).

Verification (all three plugin-dev agents previously died at the first warning with exit 1):

• agents/agent-creator.md, agents/plugin-validator.md, agents/skill-reviewer.md → all checks pass, exit 0
• A valid agent file that only triggers warnings → runs to the summary, exit 0
• An intentionally invalid file (bad name, missing color) → all 3 errors reported, exit 1

New validate-agent.test.sh next to the script covers all three cases as a regression test.

Fixed validate-agent.sh aborting at the first warning and rejecting valid agent files

🤖 Generated with Claude Code

[fsc-eriker]

Wouldn't a simpler fix be to switch to preincrement ((++warning_count))

[konsta95]

Reproduced against base 8b6ef81 and head 0989f29 (bash 5.3, GNU awk 5.3). Two notes.

On ((++warning_count)) — it stops the crash, since from 0 the increment evaluates to 1 and ((...)) returns 0. But it only covers the first of the three bugs here. Applying pre-increment alone to the base script: agents/agent-creator.md reaches the summary and exits 0, but description is still read as 244 chars instead of 1170, so the false ⚠️ should include <example> blocks still fires; and a valid agent that merely omits the optional tools: key still exits 1, aborting right after the color check, because the unguarded TOOLS=$(... | grep '^tools:' ...) propagates grep's exit 1.

Worth noting the assignment form this PR uses is unconditionally safe, where the arithmetic-command alternatives are only conditionally safe:

x=0;  ((x++))      -> exit 1     # the bug
x=0;  ((++x))      -> exit 0
x=-1; ((++x))      -> exit 1     # still aborts
x=-1; x=$((x + 1)) -> exit 0     # always safe

Counters that only climb from 0 never reach that edge, so pre-increment would work here — but count=$((count + 1)) doesn't depend on that reasoning holding.

A datapoint that might be worth adding to the PR body — the blast radius is wider than plugin-dev's own three agents. The "Complete Format" block in skills/agent-development/SKILL.md, the one this skill tells authors to copy (description ending in Examples:, then unindented <example> blocks), is itself rejected by the pre-fix validator: exit 1, description read as 73 of 506 characters, same false warning. The validator rejected the shape its own documentation prescribes. It passes cleanly after this PR.

For the two sibling validators carrying the same set -e interaction, #66416 has had a diagnosis and a three-line fix open since June — I've left the details there rather than duplicating them here.

[sylvesterkaczmarek]

description continuation only stops at name|model|color|tools, but valid agent frontmatter also includes keys such as permissionMode, disallowedTools, mcpServers, and maxTurns. Since YAML key order is free, a folded description followed by one of those keys absorbs it into DESCRIPTION and can false-flag later validation. Stop on any new top-level frontmatter key (or parse YAML) and add a regression with description: > followed by permissionMode:.

[konsta95]

Follow-up with measurements, since two things are being discussed at once here.

The absorption @sylvesterkaczmarek describes is real. With a description followed by permissionMode/maxTurns, this PR's extractor reports the description plus those keys (fixture: 121 chars reported for an 80-char description); the "too short" warning is suppressed and the "very long" warning can fire across the 5000 boundary (5024 reported for 4981 chars). #76985 has the identical name|model|color|tools stop list.

"Stop at any top-level key" is not the fix, though. Run against this plugin's own agents it stops at their unindented Context: lines: agent-creator drops from 1170 to 255 chars, plugin-validator 1182 to 285, skill-reviewer 1157 to 277. The <example> warning stays quiet only because <example> precedes Context:.

The underlying problem is that those files do not parse in Claude Code at all. On 2.1.259, copied to .claude/agents/ they log YAML frontmatter ... failed to parse and was ignored and vanish from the agent roster; loaded with --plugin-dir they log Failed to parse YAML frontmatter and are registered with the placeholder description Agent from plugin-dev plugin, no examples. Same for pr-review-toolkit's code-simplifier. Three shapes are each fatal on their own: a value ending in :, unindented continuation lines, and : inside indented continuation lines. So the extractor debate is about files the product rejects, and validate-agent.test.sh here asserts exit 0 for them. That test also cannot see a regression of the multi-line fix: reverting the extraction to the single-line form still passes 5/5, because the false <example> flag is a warning and the test checks exit codes only.

What works: description: |- block scalars with two-space indentation. Converted that way, all four files load on both paths with text equal to the file, and this PR's extractor and a stop-on-any-key extractor agree on them. I have a patch with the conversions, the matching doc/template updates in the agent-development skill, a parser-shaped extraction in validate-agent.sh that rejects the three fatal shapes, and a test that fails against the current script, against this PR's script and against a stop-on-any-key variant. Filed as #91871; the patch is on the branch fix/agent-description-block-scalars of konsta95/claude-code (pull request creation is restricted to collaborators here, so it is a branch rather than a PR).

[sylvesterkaczmarek]

Thanks, that evidence changes the right fix. I agree that simply extending the stop-key list is not sufficient. The absorption bug I reported is real, but if the plugin’s own fixtures are not valid frontmatter to Claude Code, then a validator test that treats them as valid is also testing the wrong language. I would align this validator with the product’s actual frontmatter/YAML parsing rules, or at minimum use fixtures that the product itself accepts, and then validate the extracted description. That resolves the key-order ambiguity without inventing a second parser grammar in awk.

[konsta95]

Agreed, and I've reworked it that way on the same branch, fix/agent-description-block-scalars of konsta95/claude-code, now at 9b01484 (pull request creation is restricted to collaborators here, so it is still a branch rather than a PR; the compare view merges clean against main and opens as a PR in a couple of clicks for anyone with access). The four agent files are unchanged from the original commit; the rework is validate-agent.sh, its test, and the skill doc.

Short version: the script doesn't decide parse-ability anymore. It drops the file into a throwaway plugin dir and runs claude plugin validate --json, so the verdict is the product's own parser (retry pass included) and the error text is the product's. No claude on PATH, or a report that didn't complete, means "not verified" and exit 2 rather than a pass.

What it still checks on its own, because the validator doesn't: an empty or null description (a project agent gets dropped as missing it, a plugin agent shows "Agent from plugin"), a numeric one (project agent dropped, plugin agent shows the digits), a --- anywhere inside the frontmatter (the loader cuts at the first one it sees, even mid-line, and everything after it is silently gone), and the usual style stuff (<example> blocks, "Use this agent when", lengths) on the text the plugin loader actually hands the model.

Tests: the suite runs the product once over 92 description shapes and asserts that the script's verdict is the product's verdict plus those checks of its own, nothing else. For 69 of them it also compares the extracted text with what the runtime showed in the agent listing (or, where I didn't capture a listing, a replay of the loader path through Bun). All fixtures are files the product accepts or rejects on its own terms, so that covers your "at minimum" option too.

How I checked it: no skipping — if claude is missing or too old for --json the suite fails, it doesn't go green. In all six review rounds, the expanded suite was also run against the previous commit's script and failed there. Green under gawk, mawk, busybox awk and the original BWK awk; not run on macOS. Measured on Claude Code 2.1.266 / Bun 1.4.1 — the verdicts follow whatever claude is installed, the expected texts are pinned to what 2.1.266 shows. I checked the YAML-library route too: on the original 30 shapes Bun agrees with yaml, js-yaml and PyYAML on every accept/reject, but the product's retry accepts three shapes all four reject, so a library wouldn't have matched the product. The branch was written with Claude Code, and a Codex session reviewed it over six rounds, reproducing the description-text findings against a live agent listing and the rest (report handling, large files, a false CRLF note) with focused reproducers. Last postfix review round came back clean. Two things it doesn't model, noted in the script header: an invalid UTF-8 byte (the loader turns it into U+FFFD; only the printed text differs) and a \0 escape (bash can't hold a NUL, so a description that is nothing but \0 is reported empty).

The description text still goes through a small YAML reader in the script, because --json gives the verdict but not the parsed frontmatter. If the report included that, the reader could go.

[konsta95]

Requesting changes. The count=$((count + 1)) and || true fixes are right and should land as they are. Two things block the rest, re-measured today against 0989f29 on Claude Code 2.1.268 and 2.1.270:

• The test asserts exit 0 for agents/agent-creator.md, plugin-validator.md and skill-reviewer.md, but claude plugin validate rejects all three: "YAML frontmatter failed to parse ... every other frontmatter field silently dropped." The suite pins the validator to accepting files the product discards.
• The suite cannot see the multi-line description fix regress. With the extraction reverted to the original single-line grep '^description:', all 5 tests still pass while the false "should include blocks" warning fires (244 of 1170 characters read); it checks exit codes and the summary line only.

The extractor also absorbs trailing keys: with permissionMode/maxTurns after the description it reports 112 characters for a 76-character value (inline).

A reworked version that takes the parse verdict from claude plugin validate --json and converts the four agents to description: |- block scalars is on konsta95:fix/agent-description-block-scalars (#91871), green on 2.1.268 and 2.1.270.

[konsta95]

Any key outside this list that follows description: is absorbed into it: permissionMode: default + maxTurns: 5 gives 112 reported characters for a 76-character description. Stopping at any top-level key is not the fix either, since it cuts plugin-dev's own agents at their unindented Context: lines (1170 → 255). The shape the product parses is a description: |- block scalar.

[konsta95]

These three files fail the product's parser (claude plugin validate, 2.1.268 and 2.1.270: "YAML frontmatter failed to parse"). Asserting exit 0 here pins the validator to accepting files the runtime drops.

[konsta95]

Exit code plus this line is not enough to see the description fix regress: with the single-line extraction restored, 5/5 still pass while the false <example> warning fires. A fixture with a multi-line description asserting that warning is absent would close it.