feat(isthmus): map REGEXP_EXTRACT to Substrait regexp_match_substring

[nielspardon]

What

Maps Calcite's BigQuery-library REGEXP_EXTRACT(value, regexp) operator to Substrait's two-argument regexp_match_substring(input, pattern).

Why

REGEXP_EXTRACT had no entry in FunctionMappings, so SQL using it failed to convert to Substrait (Unable to convert call REGEXP_EXTRACT(...)). The two-argument form lines up directly with the two-argument regexp_match_substring impl in functions_string.yaml.

Scope / notes

• Two-argument form only. Calcite's optional position/occurrence args (3–4 arg forms) are not handled — Substrait has no 3/4-arg impl, so those would need argument padding to the 5-arg impl and are left for a follow-up.
• Options (case_sensitivity, multiline, dotall) are defaulted automatically by the function matcher, the same way substring's negative_start option is.
• Semantic caveat: BigQuery REGEXP_EXTRACT returns capture group 1 when the pattern contains a capturing group; the 2-arg Substrait impl returns the full match. Identical for group-less patterns, divergent otherwise. Called out in a code comment.
• AutomaticDynamicFunctionMappingRoundtripTest had used regexp_match_substring as an example of an unmapped function. Since it is now mapped, the test is repointed at the still-unmapped regexp_count_substring.

Testing

• New round-trip test in StringFunctionTest over c16/vc32/vc.
• StringFunctionTest, AutomaticDynamicFunctionMappingRoundtripTest, and FunctionConversionTest pass locally.

🤖 Generated with AI

[alexandrefimov]

The semantic caveat is the whole review for me, and I think it is bigger than a code comment can carry — mostly because the entry works in both directions.

Sig feeds the reverse lookup too (FunctionConverter:97), so this also makes a Substrait regexp_match_substring(input, pattern) come back as Calcite REGEXP_EXTRACT. In that direction the plan says "the full match" and the operator it becomes means "capture group 1 if the pattern has one" — so the divergence is introduced by the conversion rather than inherited from the user's SQL. Going the other way a BigQuery user writing REGEXP_EXTRACT(s, '([0-9]+)-') gets a plan that means digits plus the dash, and nothing anywhere reports it.

Checking the YAML (functions_string.yaml, spec v0.101.0), the two-argument form is a separate function entry rather than the 5-argument one with defaults, and its description pins the semantics: "It returns the substring matching the full regular expression." The 5-argument entry is where group lives — 0 full match, 1 first capture group. So the missing dimension is group, and the 3/4-argument padding in the follow-up note would not reach it: position and occurrence are the ones padding could supply.

That also rules out the obvious repair of mapping to the 5-argument impl with group = 1, since the spec calls an out-of-range group undefined behaviour and a pattern without a capture group has no group 1. For a literal pattern the right group is decidable by looking at the pattern; for a dynamic one it is not.

Which leaves the question I would put to you rather than answer: is a plan that converts and means something else better than one that fails to convert? For most of the mappings in this list the answer is easy because they agree; here the disagreement covers what I would guess is the common use of REGEXP_EXTRACT. If it is worth landing anyway — and there is a real argument that group-less patterns are worth supporting today — I would want the divergence somewhere a user can encounter it, not only in a source comment.

[alexandrefimov]

'[0-9]+' has no capture group, so this passes through the one case where the mapping is exactly right, and nothing in the suite exercises the case the new comment warns about.

A second parameter set with a capturing pattern — '([0-9]+)-' — would at least pin that those patterns are mapped too, which is the surprising half.

[nielspardon]

The semantic caveat is the whole review for me, and I think it is bigger than a code comment can carry — mostly because the entry works in both directions.

That was the main reason why this is still a draft PR. I wasn't too happy about the drafted mapping and wanted to see what we are missing semantic-wise in the upstream spec or if we have to just deviate from being Calcite compatible here as discussed in the other issue.

[alexandrefimov]

On what is missing upstream — I do not think it is a gap in the spec. REGEXP_EXTRACT's rule is "the capture group if the pattern has one, the full match otherwise", so the meaning of a call depends on the content of one of its arguments. Substrait's five-argument regexp_match_substring refuses that: the producer names 0 or 1, and the plan then means one thing. That reads as the spec being deliberately narrower rather than incomplete — and as the same shape you described on #1116, a dialect rule Calcite carries through its core and Substrait keeps out of it.

Which leaves the deviation question, and there may be less to deviate on than it looks. Checked in the Calcite 1.42.0 jar rather than assumed: SqlFunctions$RegexFunction.regexpExtract calls checkMultipleCapturingGroupsInRegex before reading groupCount() and selecting a group by it, so the operator admits at most one capturing group — the group is always 0 or 1. And whether a pattern has one is decidable at conversion time whenever the pattern is a literal, as it is in this PR's test. So the five-argument impl with group set from the pattern is an exact mapping for literal patterns, and only a dynamic pattern would have to be refused.

That is a good deal more than the one-line entry, and it buys exactness rather than coverage, so it may well not be the trade you want. No preference from me — mostly wanted to put the "is this a spec gap" half of the question to rest.

Sorry for walking into a draft — I had not looked at the status before writing.

[nielspardon]

Sorry for walking into a draft — I had not looked at the status before writing.

All good. I also didn't state my concerns in the PR description which would have created more clarity. Thanks for taking a look and providing review input.