Skip to content

fix(isthmus)!: preserve literal nullability converting to Calcite - #1110

Merged
nielspardon merged 2 commits into
substrait-io:mainfrom
alexandrefimov:issue-1066-literal-nullability
Aug 21, 2026
Merged

fix(isthmus)!: preserve literal nullability converting to Calcite#1110
nielspardon merged 2 commits into
substrait-io:mainfrom
alexandrefimov:issue-1066-literal-nullability

Conversation

@alexandrefimov

@alexandrefimov alexandrefimov commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Substrait literal nullability did not survive conversion to Calcite: RexBuilder gives every non-null literal a NOT NULL type, so nullable literals came back non-nullable, and a homogeneous NestedList holding a nullable literal next to a nullable field reference came back mixed-typed and was rejected. Only the character/binary paths kept the type, incidentally, by picking up a cast wrapper.

This makes that shape the contract. Scalar literal visitors request their exact Substrait type (makeLiteral(value, type, allowCast = true)); the makeTimeLiteral / makeTimestampLiteral / makeIntervalLiteral paths wrap through preserveNullability. Interval-day no longer needs it: #1120 moved that visit onto the type converter, which already carries nullability. On the way back, CallConverters.CAST folds a cast over a non-null literal that changes nothing but nullability into the literal itself, carrying the cast's nullability, instead of surfacing an Expression.Cast. LogicalValues tuples keep bare literals — the row type already declares field nullability there, so that one cast is unwrapped, and only that one: a cast doing anything else is doing work, and dropping it would leave the literal under a row type that does not describe it.

Found along the way and left as is, tracked separately: the Calcite→Substrait path normalizes time/timestamp literals to microsecond precision (#1114) and a precision_timestamp_tz literal returns as a plain precision_timestamp (#1113).

Closes #1066

BREAKING CHANGE: Substrait→Calcite conversion now represents a nullable literal as a cast of the literal to its nullable type where it previously produced a bare literal with a non-nullable type, and converted row types become nullable accordingly. Calcite→Substrait conversion folds nullability-only casts over literals into nullable literals instead of Expression.Cast.

@nielspardon

Copy link
Copy Markdown
Member

With #1120 merged you can update this PR and we can check what needs fixing.

@alexandrefimov
alexandrefimov force-pushed the issue-1066-literal-nullability branch from 188a3b3 to 85a5996 Compare August 19, 2026 12:43
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Rebased. The interval-day half went away rather than merged: #1120 moved visit(IntervalDayLiteral) onto makeLiteral(value, typeConverter.toCalcite(...), true), so the type already carries the nullability there and the preserveNullability wrapper had nothing left to do. What remains is makeTimeLiteral, makeTimestampLiteral and the year-interval makeIntervalLiteral.

The round-trip table keeps its interval_day row. It passes because of #1120 now rather than because of this PR, but it seemed worth keeping as coverage rather than deleting.

Also checked this against the neighbours: #1110, #1121 and #1127 all touch the same visits, and a tree with all of them merged onto main builds. Not something CI can tell us, since it only ever sees them one at a time.

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase reasoning checks out — visit(IntervalDayLiteral) on main does go through the type converter now, so that half really is subsumed rather than merged. Design verified against Calcite 1.42: makeAbstractCast does no literal folding, so the nullable shape is stable and matches what the scalar visitors produce. Three small things inline.

Separately, the precision_timestamp_tz degradation and the microsecond normalization are recorded only in a test comment — worth issues so they don't get lost.

Calcite types every non-null RexLiteral NOT NULL, so a nullable
Substrait literal came back non-nullable and a homogeneous NestedList
of a literal and a field reference came back mixed-typed and was
rejected. Convert nullable scalar literals as CAST(literal AS nullable
type) - the shape the character paths already produced - and fold the
nullability-only cast back into a nullable literal on the return trip.
LogicalValues tuples keep bare literals: the row type already declares
field nullability there.
The tuple unwrap stripped any cast over a literal, so a cast doing real
work — a varchar literal longer than its declared length, which Calcite
cannot fold — was dropped and the literal landed under a row type that
does not describe it. It used to fail loudly there, and does again.

The literal fold gets the same treatment for the same reason: an exact
no-op cast is not the nullability representation, so it stays a cast.
Neither direction can produce one today, since Calcite drops an exact
cast over a literal, but the guard now says what the contract says.

Plus a test on the cast shape itself, which both the fold and the
unwrap key on and nothing pinned.
@alexandrefimov
alexandrefimov force-pushed the issue-1066-literal-nullability branch from 85a5996 to 30234ee Compare August 19, 2026 14:49
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Rebased. The core commit is gone: #1136 landed the same one-line fix, and its LiteralTypeDerivationTest covers what the test in that commit covered plus interval_day and interval_compound, so keeping mine would only have duplicated it. The description and the breaking footer no longer mention it.

On the two things recorded only in a test comment — both are filed already, and both are closed by #1121: #1114 for the microsecond normalization, whose interval third went with #1120, and #1113 for the precision_timestamp_tz degradation. The description points at them now instead of leaving them in a comment.

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nielspardon
nielspardon merged commit 3b46130 into substrait-io:main Aug 21, 2026
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

isthmus: literal nullability is dropped converting to Calcite, producing non-conformant plans

2 participants