Skip to content

Emit a line break before rescue after an omitted keyword argument value - #10

Open
tarebyte wants to merge 2 commits into
masterfrom
fix/hash-shorthand-before-rescue
Open

Emit a line break before rescue after an omitted keyword argument value#10
tarebyte wants to merge 2 commits into
masterfrom
fix/hash-shorthand-before-rescue

Conversation

@tarebyte

@tarebyte tarebyte commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Summary

A value-omitted keyword argument (foo key:, Ruby 3.1+) may take its value from the following line, so the pair rule offers the _no_line_break external token as a hint and the scanner responds by withholding the line break that would otherwise terminate the statement.

rescue can never begin a value, but with no line break emitted the parser preferred rescue_modifier — whose body and keyword must share a logical line — over the rescue clause of the enclosing body. So this:

begin
  key = 10
  foo key:
rescue => e
end

parsed as a rescue modifier wrapping the call, with => e reported as an ERROR. Prism parses it as a BeginNode with a foo(key:) call and a rescue clause.

The grammar declares no conflicts, so the generated parser resolves that preference statically and always takes the modifier path. Expressing the distinction in grammar.js would mean the grammar deciding on a token the scanner has not emitted yet, so the fix is made where the lookahead already lives: the scanner now records when it withholds a line break and, if the next token turns out to be the rescue keyword, emits the line break after all. The identifier lookahead used for that check is shared with the existing hash-key / identifier! scanning so words that merely start with r keep working.

The withheld-line-break flag is deliberately not serialized. It is cleared at the top of every whitespace scan, set only while that scan is consuming whitespace, and consumed later in the same scan() call, so it never has to survive across a serialize boundary.

grammar.js, src/grammar.json and src/parser.c are unchanged — this is a scanner-only fix.

Validation

  • 335 corpus parses and 5 highlight files, zero failures.
  • Parsing all 4,437 ruby/spec files with master and with this branch produces byte-identical trees; the only diff lines are parse-timing numbers. This is a regression check rather than evidence for the fix: nothing in that corpus reaches the changed path.
  • Prism conformance: 4,431 valid files, 0 expected mismatches, 0 unexpected mismatches. That count excludes 6 of the 4,437 files — 2 that Prism itself rejects and 4 with unsupported encodings.
  • Incrementally editing a next-line value into rescue => e and back again produces trees identical to fresh parses in both directions.

Covered by new corpus tests:

  • rescue on the same line still parses as rescue_modifier (x = compute rescue nil, y = foo(1) rescue nil).
  • Next-line values beginning with r still parse as values: rescue_handler, reload!, r, Regexp.

Checked by direct parsing, not by checked-in corpus cases: rescue clauses in begin, def, class and do blocks, plus multiple rescue clauses with ensure, all parse with no error nodes.

One pre-existing corpus case was renamed and its expected tree updated from rescue_modifier plus ERROR to the corrected call-and-rescue tree.

References

Addresses the foo key: before rescue case reported in tree-sitter#237 — one of the two repros @aibaars reduced in that issue. The other repro, ... 3, already parses as a beginless range on master and is already covered there by test/corpus/declarations.txt:577; no new test is needed for it.

tarebyte and others added 2 commits August 24, 2026 02:18
…alue

A value-omitted keyword argument (`foo key:`, Ruby 3.1+) may take its value
from the following line, as in `foo key:\n  bar`. To allow that, the pair rule
offers the `_no_line_break` external token as a hint, and the scanner responds
by withholding the line break that would otherwise terminate the statement.

The `rescue` keyword can never begin a value, so withholding the line break
before it left the parser in a state where `rescue_modifier` — which requires
its body and the keyword to be on the same logical line — was preferred over
the `rescue` clause of the enclosing body. `begin ... foo key: ... rescue => e
... end` therefore parsed as a rescue modifier wrapping the call, with the
`=> e` reported as an ERROR. The grammar declares no conflicts, so this choice
is made statically at generation time and cannot be corrected in `grammar.js`;
the only way to distinguish the two is to give the parser the line break.

The scanner now records when it withholds a line break and, when the next token
turns out to be the `rescue` keyword, produces the line break after all. The
identifier lookahead used for that check is shared with the existing hash-key /
`identifier!` scanning so that words merely starting with `r` keep working.

Fixes tree-sitter#237

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ac21705f-befb-4606-9c83-657e3e9ea0dc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant