Detailed debug info for reduce/reduce shift/reduce errors?

[DavidHVernon]

I was wondering if there is a way to get more detailed info on CTConflictsError errors (reduce/reduce and shift/reduce). In particular it would be handy to know which production rules are clashing.

[ratmice]

have a look at the conflicts function of lrpar,
You may need to error_on_conflicts(false), and you should be able to get more detailed info from there.

From my lsp here it gets the prod idx then converts that to spans & lsp errors. It is a bit messy though.
https://github.com/ratmice/nimbleparse_lsp/blob/main/server/src/parse_thread.rs#L505-L571

[DavidHVernon]

Thank you very much.

[ratmice]

It seems worth noting that code uses the RTParserBuilder, but I think CTParserBuilder should behave basically the same in this regard. Also worth mentioning I don't think that code has made it to a published version of the lsp yet, but should basically work.

[DavidHVernon]

I'm having trouble seeing how to get access to an lrpar, and where to use conflicts. I don't know much about these libraries (because they work well enough that I haven't needed to), but I get that this needs to be in the build phase.

I have copied the boilerplate build.rs code. It looks like the parser is being built by the lexer builder via lrpar_config(). That is where I get lost.

[ratmice]

I see what you mean, I think there are 2 ways you can go about it.
In the old style, you build the CTLexerBuilder, CTParserBuilder separately, and combine them with a
token map, much like is described in https://softdevteam.github.io/grmtools/master/book/manuallexer.html
So you can build the CTParser yourself, outside of CTLexerBuilder, the example I can find from the repository that behaves this way is

grmtools/lrpar/cttests/src/cgen_helper.rs

Lines 91 to 94 in 4ed68d6

	let mut cp_build = CTParserBuilder::<DefaultLexerTypes<u32>>::new()
	.yacckind(yacckind)
	.grammar_path(pg.to_str().unwrap())
	.output_path(&outp);

grmtools/lrpar/cttests/src/cgen_helper.rs

Lines 109 to 112 in 4ed68d6

	let mut cl_build = CTLexerBuilder::new()
	.rule_ids_map(cp.token_map())
	.lexer_path(pl.to_str().unwrap())
	.output_path(&outl);

There is some conflict testing code here also (build calls on lines 104-140 respectively)
https://github.com/softdevteam/grmtools/blob/4ed68d6feeff01738c073eced73a3192daad51c7/lrpar/src/lib/ctbuilder.rs#L1363C5-L1370

In theory, you could also construct a CTParserBuilder::new() set the output path out of the way, and .build().conflicts(), produce your errors, and panic. That really seems a bit hacky especially with the output paths part. I'd recommend the way above first.

[ratmice]

There is still one issue to be resolved, which is errors_on_conflicts(false) is going to allow the CTParserBuilder to generate output.

So, maybe the right way, is to do it in two steps:

• Contruct a LRNonStreamingLexer::new and RTParserBuilder checking for conflicts and erroring,
• if that succeeds build a CTLexerBuilder as normal?

I think that might be the most robust way to achieve this currently.

[ratmice]

Focusing on ways that we might improve this in the future:

The first thing that comes to mind is:

• CTLexerBuilder could gain an fn on_lex_build_error(f: fn(LexBuildError) -> Result<(), Box<dyn Error>>)
• CTParserBuilder could gain an fn on_grammar_error() -> (f: fn (cfgrammar::YaccGrammarError) -> Result<(), Box<dyn Error>>)
• Also a fn on_conflicts(f: fn (...) -> Result<(), Box<dyn Error>>)

These return values would then escape through the Result<..., Box<dyn Error>> result from CTLexer::build().
Curious what @ltratt thinks about the above sketch

[DavidHVernon]

@ratmice @ltratt As a user I would be tickled pink if in the build.rs it was as simple as...

cfgrammar::yacc::generate("y", "l");

And in main.rs

cfgrammar::yacc::parse_str("");

And detailed messages on conflicts (and other things) were output when/if they occur.

And yes, I'm sure this is rather naive.

[ltratt]

I must admit that my memory of these aspects is a bit fuzzy, but I do remember some of the challenges:

1. Yacc's default is to warn about conflicts, but to resolve them based on some (fairly arbitrary) rules. This is used in some real grammars (e.g. Lua's), so we need to support it. errors_on_conflicts(false) was intended for these grammars.
2. In general, I tend to think that relying on Yacc's conflict resolution is a bad idea, so grmtools defaults to erroring out by default. You can also get it to print out information about the state table etc. (somehow? do you need to use nimbleparse for that? I don't remember!). However, personally I am too stupid to be able to understand the statetable of any grammar longer than 3 or 4 lines.

I don't necessarily object to finer-grained information, but I can't immediately see how people might end up using it. In a better, parallel univerise, Yacc would never have gifted us default conflict resolution....

[DavidHVernon]

@ltratt I really like that this library DOESN'T comply with the arbitrary yacc rules - it is forcing me to understand the real underlying issues. It is painful, but the only reason that I am writing a language is to learn to write languages.

[DavidHVernon]

@ratmice

I'm getting closer. I would paste my code in here but I am having trouble formatting it well (I don't collaborate on github much).

My issue now is that when I build the CTParser up front the CTParserBuilder.build() returns an error (CTConflictsError{0 Reduce/Reduce, 1 Shift/Reduce}), but I need the CTParser to get to the detailed conflicts() info.

What am I missing?

I can parr things down and publish it in a repo if that helps.

[ratmice]

@DavidHVernon that is where error_on_conflicts(false) comes into play suppressing the CTConflictsError.
After that you should be able to get a CTParser and if there are conflicts rm generates_parser_output.rs presumably.

[DavidHVernon]

@ltratt Yep! You said that earlier and it didn't sink in. Thanks again. I can now see the problem in my grammar.