parse: merge fn syntax + cleanup item parsing

[Centril]

Here we continue the work in #67131 in particular to merge the grammars of fn items in various positions.

A list of language level changes (as sanctioned by the language team in #65041 (comment) and #67131):

• self parameters are now syntactically allowed as the first parameter irrespective of item context (and in function pointers). Instead, semantic validation (ast_validation) is used.

• Syntactically, fn items in extern { ... } blocks can now have bodies (fn foo() { ... } as opposed to fn foo();). As above, we use semantic restrictions instead.

• Syntactically, fn items in free contexts (directly in a file or a module) can now be without bodies (fn foo(); as opposed to fn foo() { ... }. As above, we use semantic restrictions instead, including for non-ident parameter patterns.

• const extern fn feature gating is now done post-expansion such that we do not have conditional compatibilities of function qualifiers in parsing.

• The FnFrontMatter grammar becomes:

  Extern = "extern" StringLit ;
  FnQual = "const"? "async"? "unsafe"? Extern? ;
  FnFrontMatter = FnQual "fn" ;

  That is, all item contexts now syntactically allow const async unsafe extern "C" fn and use semantic restrictions to rule out combinations previously prevented syntactically. The semantic restrictions include in particular:

  • fns in extern { ... } can have no qualifiers.
  • const and async cannot be combined.

• To fuse the list-of-items parsing in the 4 contexts that items are allowed, we now must permit inner attributes (#![attr]) inside trait Foo { ... } definitions. That is, we now allow e.g. trait Foo { #![attr] }. This was probably an oversight due to not using a uniform parsing mechanism, which we now do have (fn parse_item_list). The semantic support (including e.g. for linting) falls out directly from the attributes infrastructure. To ensure this, we include a test for lints.

Put together, these grammar changes allow us to substantially reduce the complexity of item parsing and its grammar. There are however some other non-language improvements that allow the compression to take place.

A list of compiler-internal changes (in particular noting the parser-external data-structure changes):

• We use enum AllowPlus/RecoverQPath/AllowCVariadic { Yes, No } in parser/ty.rs instead of passing around 3 different bools. I felt this was necessary as it was becoming mentally taxing to track which-is-which.

• fn visit_trait_item and fn visit_impl_item are merged into fn visit_assoc_item which now is passed an AssocCtxt to check which one it is.

• We change FnKind to:

  pub enum FnKind<'a> {
      Fn(FnCtxt, Ident, &'a FnSig, &'a Visibility, Option<&'a Block>),
      Closure(&'a FnDecl, &'a Expr),
  }

  with:

  pub enum FnCtxt {
      Free,
      Foreign,
      Assoc(AssocCtxt),
  }

  This is then taken advantage of in tweaking the various semantic restrictions as well as in pretty printing.

• In ItemKind::Fn, we change P<Block> to Option<P<Block>>.

• In ForeignItemKind::Fn, we change P<FnDecl> to FnSig and P<Block> to Option<P<Block>>.

• We change ast::{Unsafety, Spanned<Constness>}> into enum ast::{Unsafe, Const} { Yes(Span), No } respectively. This change in formulation allow us to exclude Span in the case of No, which facilitates parsing. Moreover, we also add a Span to IsAsync which is renamed to Async. The new Spans in Unsafety and Async are then taken advantage of for better diagnostics. A reason this change was made is to have a more uniform and clear naming scheme.

  The HIR keeps the structures in AST (with those definitions moved into HIR) for now to avoid regressing perf.

• Various cleanups, bug fixes, and diagnostics improvements are made along the way. It is probably best to understand those via the diffs.

I would recommend reviewing this commit-by-commit with whitespace changes hidden.

r? @estebank @petrochenkov

Other comments:

Please avoid "fns" in error messages, e.g. "fn in extern blocks" or "only allowed in associated fns", use "functions" instead, or at least "fn items".

---

Functions taking multiple bools could use bitflags instead of enums, i.e. Foo::Yes, Bar::Yes -> Foo | Bar.

---

The FnFrontMatter grammar becomes:

I don't see default in this list.
Looks like it's not supported in non-trait positions, so the parsing is not fully unified yet?

---

This change in formulation allow us to exclude Span in the case of No, which facilitates parsing.

The No span was supposed to be used as an insertion point for missing qualifiers, but if it's not used, then I'm ok with removing it.

---

Looks like we are very close to using a single routine like parse_mod_items for all kinds of item containers (modules, traits, impls, extern blocks, but not expression blocks yet, that'll perhaps come later).
Then we'll be able to just filter away all items that are not fn/const/type/macro-call and report much much better errors for situations like

trait Tr {
    static S: u8 = 0;
}

which currently look pretty bad in tests.

Meta: this PR also was too big to be review-able carefully.
I understand that it may require some double work, but some changes from this PR are mechanical or don't affect the logic, perhaps they could be split into a separate PR from changes affecting logic.
But maybe it's too late because I already reviewed it.

I made sure that all commits passed UI tests so it shouldn't be hard to take chunks and move them into separate PRs (with comments addressed). I'll try to do that in way that seems sensible.

Functions taking multiple bools could use bitflags instead of enums, i.e. Foo::Yes, Bar::Yes -> Foo | Bar.

I would prefer avoiding this as these enums are mostly unrelated flags. Merging them would result in making it harder to track what state is passed in and whatnot. Part of the reason for this refactoring was so that possibly getting rid of some of the flags would be facilitated and bitflags would make it harder than bools.

I don't see default in this list.
Looks like it's not supported in non-trait positions, so the parsing is not fully unified yet?

The front matter is only the "effect qualifiers" (intrinsic to functions themselves, and unrelated to other item forms) + the fn keyword. I'm not going beyond fns in merging right now. defaultness is a property of associated items so I would need to unify types and consts too. I think we need some design discussion re. whether to syntactically allow default mod x { ... } or whether defaultness should be moved into each variant that cares about it.

The No span was supposed to be used as an insertion point for missing qualifiers, but if it's not used, then I'm ok with removing it.

Didn't seem like they were used, but we can also recover insertion points from the start / ends as the order is fixed.

All remaining review comments have been addressed.
I'd recommend reviewing the remainder with white-space changes ignored.

@bors r+

📌 Commit ad72c3a has been approved by petrochenkov