Fix Unicode symbol scanning - #293
Open
ydah wants to merge 1 commit into
Open
Conversation
This was referenced Aug 23, 2026
tarebyte
added a commit
to tarebyte/tree-sitter-ruby
that referenced
this pull request
Aug 24, 2026
The parser diverged from Ruby on a number of constructs that ruby/spec
exercises, which is why the Prism conformance gate shipped with an
allowlist. This closes those gaps and empties the allowlist.
Scanner
Tokens that share a prefix with another reading are now scanned
externally so surrounding context can choose between them: `+`, `<<`,
and `&` as unary or binary operators, `%=` as a modulo assignment rather
than the opening of a percent literal, and `/` as a regex that begins a
range endpoint. Comments moved to the scanner as well, where `=begin`
opens a block comment only when the keyword ends the token, so an
identifier such as `=beginning` is left alone.
Heredocs previously tracked "the next body to start" and "the body being
scanned" with a single cursor. Those refer to the same heredoc until
heredocs nest, at which point the enclosing body was matched against the
nested terminator and the outer terminator was left over as a stray
expression. Started heredocs are now kept as a prefix, innermost last,
followed by the pending ones in the order they were opened. A heredoc
opened inside a running body records how many bodies were active at that
point, so it suspends the body it was opened inside of and never
interrupts a sibling that is already running.
Grammar
A lambda's parameter list is no longer shared with a method's. It
accepts block-local declarations, as in `->(a; b) { }`, and a bare
default that is a call does not swallow the block that opens the lambda
body. An endless method body is modeled on parse.y's `endless_command`,
so its arguments are `arg`s rather than the broader `expr`s a command
argument list allows, which keeps the enclosing `def` from being
reparsed as a call receiver. A dangling range operator now continues
onto the following line and nests, matching Ruby. Comments that precede
a declaration body belong to that body, and `return` takes its own
argument list so a parenthesized value parses as one.
CI
Examples are parsed recursively rather than only at the top level, the
parser test ABI is pinned to 15, and the Prism allowlist is empty.
References
Fixes:
tree-sitter#148
tree-sitter#203
tree-sitter#218
tree-sitter#242
tree-sitter#244
tree-sitter#256
tree-sitter#272
tree-sitter#280
tree-sitter#281
tree-sitter#284
tree-sitter#290
tree-sitter#292
Reimplements:
tree-sitter#285
tree-sitter#291
tree-sitter#293
tree-sitter#294
tree-sitter#295
Fixes the range case in tree-sitter#275; that issue
also asks for `bool = not true` to be rejected, but Ruby accepts it.
Adapts the endless-method command body from
tree-sitter#282.
Validated against 337 corpus and highlight tests, 4,433 recursively
parsed example files, and 4,431 Prism-valid ruby/spec files with zero
mismatches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #292