Fix Ruby grammar conformance gaps - #4
Merged
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d800609e-6d1c-43d3-8811-d59ad9495420
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d800609e-6d1c-43d3-8811-d59ad9495420
There was a problem hiding this comment.
🟡 Changes recommended
The new comment scanner and endless-method command rules still mishandle valid Ruby inputs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Improves Ruby grammar conformance across literals, operators, declarations, control flow, and CI validation.
Changes:
- Resolves scanner ambiguities involving heredocs, comments, regexes, percent literals, and operators.
- Expands grammar handling for lambdas, ranges, returns, and endless methods.
- Strengthens corpus, ruby/spec, and recursive example validation.
File summaries
| File | Description |
|---|---|
grammar.js |
Updates Ruby grammar rules. |
src/scanner.c |
Extends external token scanning. |
src/grammar.json |
Regenerates grammar metadata. |
src/node-types.json |
Regenerates node definitions. |
test/corpus/literals.txt |
Adds literal and heredoc coverage. |
test/corpus/expressions.txt |
Adds expression and lambda coverage. |
test/corpus/declarations.txt |
Covers declarations and endless methods. |
test/corpus/control-flow.txt |
Covers returns and rescue boundaries. |
test/corpus/comments.txt |
Covers comment scanning behavior. |
test/prism/ruby-spec-allowlist.txt |
Clears resolved conformance exceptions. |
.github/workflows/ci.yml |
Pins ABI 14 and recursively parses examples. |
package.json |
Adds the grammar lint command. |
Review details
- Files reviewed: 12/13 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
MRI's grammar defines an endless method body as `endless_command : command`,
reusing the same `command` nonterminal (and therefore the same `command_args`)
as any other command call. The previous rule instead enumerated an allowlist of
argument types, which is wrong by construction: it both rejects valid programs
and, worse, silently misparses them.
The silent misparses are the more serious failure. When the allowlist runs out
of usable tokens, the command call is truncated there and the surrounding `arg`
grammar reabsorbs the remainder, yielding an error-free but incorrect tree. For
example `def f = puts 1..2` grouped as `(def f = puts 1)..2`, and
`def f = puts %w[a]` parsed the literal as a binary modulo. Because nothing in
the tree is an ERROR or MISSING node, the conformance suite reports no problem.
`command_args : call_args`, and the elements of `call_args` are `arg`s rather
than `expr`s, so the argument rule is anchored at `_arg`. That is both the
correct boundary (an endless body cannot contain a bare `and`/`or`/`in`) and the
one that resolves the ambiguity with `def` appearing as a call receiver, since
`method` is reachable from `_primary`.
Also fixed alongside:
- Lambdas accept block-local declarations. `f_larglist` is
`'(' f_largs opt_bv_decl ')'`, so `->(a; b) { }` is valid even though a
method's parameter list has no such clause.
- `!@` is a valid operator method name in `op`, next to `+@`, `-@` and `~@`.
- `=begin` only opens a block comment when the keyword ends the token;
`=beginning` was being swallowed as a comment through end of line.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b691b14c-5c15-4011-97d3-721070793751
A heredoc opened inside a running heredoc body suspends that body: Ruby reads the new body at the next line break and resumes the enclosing one once the nested terminator is reached. `open_heredocs` modelled this with a single cursor at index 0, which had to serve both "next body to start" and "body currently being scanned". Those diverge as soon as heredocs nest, so the enclosing body was matched against the nested word and its own terminator was left over as a stray expression. Split the two roles. Started heredocs are kept as a prefix, innermost last, followed by the pending ones in the order they were opened. Content scanning uses the innermost started heredoc, while a pending heredoc starts at the next line break. Each pending heredoc records how many bodies were active when it was opened so that it suspends only the body it was opened inside of, rather than interrupting a sibling that is already running. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7c65d880-b26f-4f6a-9862-8d7c1a9ff0c7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
References
defwith a command call as right-hand side tree-sitter/tree-sitter-ruby#242, fixes Parse error in anonymous block arguments tree-sitter/tree-sitter-ruby#244, fixes bug: Unary plus should parse the same way as unary minus. tree-sitter/tree-sitter-ruby#281, and fixes bug: Parser error when forwarding an anonymous block to another function without surrounding with parenthesis (ampersand &) tree-sitter/tree-sitter-ruby#290.0..now continues onto the next line and nests, matching Ruby. That issue also asks forbool = not trueto be rejected, but Ruby accepts it, so it stays accepted.string_contenttree-sitter/tree-sitter-ruby#285, Allow for more valid method calls with block arguments tree-sitter/tree-sitter-ruby#291, Fix Unicode symbol scanning tree-sitter/tree-sitter-ruby#293, Reject multiple values in parenthesized return tree-sitter/tree-sitter-ruby#294, and Support percent string equals delimiter tree-sitter/tree-sitter-ruby#295.Validation