test(docs): fail when generated docs are out of date - #4360
Merged
Conversation
Refactor the doc generators to render to memory (generatedDocFiles) and add a test that compares the rendered output against the committed files. This catches contributors who change a config option or CLI command but forget to run `make cli-docs`. Also regenerates server_options.rst, which had drifted from the policy.authzen.endpoint flag description. Assisted by AI
reinkrul
requested review from
Dirklectisch,
JorisHeadease,
gerardsn,
stevenvegt and
woutslakhorst
as code owners
June 18, 2026 08:23
Contributor
1 new issue
|
Contributor
|
Coverage Impact ⬆️ Merging this pull request will increase total coverage on Modified Files with Diff Coverage (1)
🤖 Increase coverage with AI coding...🚦 See full report on Qlty Cloud » 🛟 Help
|
; Conflicts: ; docs/generate_docs.go ; docs/pages/deployment/server_options.rst
paths-filter excluded all of docs/ (including docs/generate_docs.go and its tests) from the "code" filter that gates the test job, so the new TestGeneratedDocsAreUpToDate added in this PR never actually runs in CI. Its some-with-excludes mode makes an exclusion final, so a blanket '!docs/**' can't be selectively re-included by a later positive pattern - the exclude has to be scoped to the non-Go subpaths instead. Assisted by AI
stevenvegt
approved these changes
Sep 9, 2026
stevenvegt
left a comment
stevenvegt
left a comment
Member
There was a problem hiding this comment.
LGTM, one question about the filter.
The per-subpath exclude list duplicated the docs/ directory layout into the workflow config. Since all Go files live directly under docs/, a single '!docs/*/**' (excluding anything nested under a docs subdirectory) achieves the same result: it still treats a handful of rarely-changed root-level non-Go files (Dockerfile, conf.py, etc.) as code, which is an acceptable false-positive risk in exchange for not having to keep the filter in sync with docs/'s contents. Assisted by AI
reinkrul
added a commit
that referenced
this pull request
Sep 9, 2026
* test(docs): fail when generated docs are out of date Refactor the doc generators to render to memory (generatedDocFiles) and add a test that compares the rendered output against the committed files. This catches contributors who change a config option or CLI command but forget to run `make cli-docs`. Also regenerates server_options.rst, which had drifted from the policy.authzen.endpoint flag description. Assisted by AI * ci: run the Go test job on docs/ Go files too paths-filter excluded all of docs/ (including docs/generate_docs.go and its tests) from the "code" filter that gates the test job, so the new TestGeneratedDocsAreUpToDate added in this PR never actually runs in CI. Its some-with-excludes mode makes an exclusion final, so a blanket '!docs/**' can't be selectively re-included by a later positive pattern - the exclude has to be scoped to the non-Go subpaths instead. Assisted by AI * ci: simplify docs/ code filter to !docs/*/** The per-subpath exclude list duplicated the docs/ directory layout into the workflow config. Since all Go files live directly under docs/, a single '!docs/*/**' (excluding anything nested under a docs subdirectory) achieves the same result: it still treats a handful of rarely-changed root-level non-Go files (Dockerfile, conf.py, etc.) as code, which is an acceptable false-positive risk in exchange for not having to keep the filter in sync with docs/'s contents. Assisted by AI (cherry picked from commit dac8093)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Contributors who add/change a config option or CLI command must run
make cli-docs, but it's easy to forget — the docs then drift from the code. This adds a unit test that regenerates the docs in-memory and fails if they differ from what's committed, so CI catches it.Changes
docs/generate_docs.go: split the generators into render-to-memory functions (generatedDocFilesreturnspath → content);generateDocs()now just writes that map to disk. Output is byte-identical to before.TestGeneratedDocsAreUpToDate: comparesgeneratedDocFiles()against the committed files; on mismatch fails with"<file> is out of date; run \make cli-docs` and commit the result"`.server_options.rst, which had drifted from thepolicy.authzen.endpointflag description (needed for the new test to pass).Scope
cli-reference.rst,server_options.rst,server_options_didnuts.rst.README.rst— that's assembled by the separaterst_includestep inmake cli-docs, not by the Go generator.normalizeDefaultValue; verified the generator output is stable across runs.🤖 Assisted by AI