Extend rustdoc to accept the same set of command line flags as rustc

[ssalb1z]

This is intended to make it easier to use the same flag-set for both rustc and rustdoc in build systems without having invocations fail.

This is accomplished by sharing rustc_optgroups() between rustc and rustdoc, and adding test coverage to ensure that the two stay in sync. Rustdoc-specific flags are added on top of the rustc_optgroups table. Certain incompatible flags are not inherited from rustc_optgroups and redeclared.

As a result, -l, -O, and -g now reach doctest compilation.
Ordering of forwarded rustc flags is preserved precisely to maintain order-dependent flag semantics.

Some caveats and known limitations:

• There are some overlapping flags with differing semantics between rustc and rustdoc (e.g. -o/--out-dir)
• In most cases during documentation builds, rustc-specific flags are simply inert when passed to rustdoc. This could create some confusion e.g. with flags like -j. Not included in this change is warning on such inert flags that are only accepted for compatibility reasons.
• The help text has been overridden in a few cases to explain more clearly how the flag is interpreted in rustdoc, but in most cases the help text from rustc_optgroups() is inherited verbatim.
• --json will now allow repeated values, accumulating options across them as rustc does.
• -l has to be handled specially for compatibility with merged-doctest bundles - in that mode, the flag is passed only to the rlib compilation, and not (in duplicate) again to the final merged binary runner, which could cause errors due to duplicate symbols

Created with the help of an LLM (Claude Fable 5.1), which:

• Answered questions about the interaction of various flags used across rustc and rustdoc
• Assisted with writing some of the mechanical pieces of logic and wire-up (all help-text was still written by hand)
• Assisted in setting up the basic test harnesses (all tests were still reviewed by me by hand)
• Reviewed this PR before it was posted

This PR aims to resolve #153318
and partially resolve #43031

[fmease]

I've only skimmed the PR descripion+contents.

Do I understand correctly that the main / only motivation for this is doctests (for which we have the unstable --doctest-build-arg to forward rustc args btw)?

Furthermore, does this PR makes rustdoc blindly accept any rustc option (unless shadowed) even in non-doctest modes (html, json, freestanding markdown)? Does it mean that in non-doctest modes rustdoc accepts these flags, passes them to rustc driver without any guarantee that they're later processed / honored in any way?

If so that would be hazardous IINM. When rustc devs implement &/ stabilize "rustc" flags, T-rustdoc won't be aware of that but it would affect rustdoc's interface, too, unbeknownst to people working on the compiler. So whenever that happens it's almost inevitable that rustdoc silently accepts & ignores a new flag. Then, when it gets noticed months after stabilization & people scramble to also implement or destabilize it in rustdoc, it has a risk to break stable users.

For example, in your PR you've made --explain <CODE> legal but it just gets ignored. I'm not a fan of that.

View changes since this review

[ssalb1z]

Hello - thank you very much for looking at this, I really appreciate it!

I've only skimmed the PR descripion+contents.

Do I understand correctly that the main / only motivation for this is doctests (for which we have the unstable --doctest-build-arg to forward rustc args btw)?

Partially, but the bigger motivation is to make the command more uniform with rustc for integration into build systems easier. For example, the chromium team is trying to integrate it into their ninja builds (https://chromium-review.googlesource.com/c/chromium/src/+/7577035) - and running into challenges because the flags need to be specialized. I guess fixing this for that use-case may be possible via --doctest-build-arg as you suggested already though - although it would create non-trivial complexity in any downstream build system IMO that has to parse and forward the rustc flag combinations appropriately.

Furthermore, does this PR makes rustdoc blindly accept any rustc option (unless shadowed) even in non-doctest modes (html, json, freestanding markdown)? Does it mean that in non-doctest modes rustdoc accepts these flags, passes them to rustc driver without any guarantee that they're later processed / honored in any way?

I don't believe so, since rustc session is built with an explicit allowlist of forwarded flags (https://github.com/rust-lang/rust/pull/163056/changes#diff-ddf0fd28685f9903c26d1ffd7f1ca976f1bb88c628de453400ac7c7ed28b4d82R996-R1002), in non-doctest modes any novel args are not forwarded into the session - but maybe this is what you really were asking, yes would become accepted by the overarching binary rustdoc indeed, even if those flags remain inert when invoking the rustc driver.

If so that would be hazardous IINM. When rustc devs implement &/ stabilize "rustc" flags, T-rustdoc won't be aware of that but it would affect rustdoc's interface, too, unbeknownst to people working on the compiler. So whenever that happens it's almost inevitable that rustdoc silently accepts & ignores a new flag. Then, when it gets noticed months after stabilization & people scramble to also implement or destabilize it in rustdoc, it has a risk to break stable users.

For example, in your PR you've made --explain <CODE> legal but it just gets ignored. I'm not a fan of that.

I think that's reasonable - I can see a couple different approaches if you believe this is fundamentally an incorrect goal (of having rustc and rustdoc have a single shared target of accepted flags at the top-level).

One option could be to share some of the code but have some kind of opt-in or opt-out switch like --shadow-rustc-args or something that permits the flag forwarding natively, instead of requiring --doctest-build-arg construction (Or --strict-rustdoc-args or similar if we want the forwarding behaviour to be on by default - either way would work I believe).

Another option could be to explicitly state that the flag combinations are not expected to be cross compatible across the tools - and move forward with a handful of very specific forwarded flags (I guess that's what we did for -C and -Z), maybe I could add the forwarding for -l and we'd call it a day.

Related to the above, I was hoping to build something a bit more generic, but one option is to share some more of the options code as I have implemented here but have an allowlist of rustc forwarded flags that are permitted in rustdoc explicitly, rather than a denylist (as is currently implemented through the overrides filter). That would allow rustdoc to be explicit about when flags graduate from unstable -> stable, for example, independently of rustc, or if they would be permitted at all in any way, without having to re-implement the flag in two places each time. It would also allow us to more easily prevent accidentally having the same flag name appearing in both tools with differing semantics - which I think would be a useful property again for integration with build systems that may wish to assume otherwise.

Alternatively, if you have other suggestions or ideas for how the related issues (#153318 & #43031) should be resolved going forward, I'm more than happy to help execute those ideas as well.

Thank you again for taking the time to review this change and talk to me! Again, I really appreciate it.
I've not made any changes to this draft PR yet - I thought it would be best to agree on a shared direction before proceeding.

Cheers,
Syed

View changes since this review