fix(isthmus)!: preserve literal nullability converting to Calcite - #1110
Conversation
|
With #1120 merged you can update this PR and we can check what needs fixing. |
188a3b3 to
85a5996
Compare
|
Rebased. The interval-day half went away rather than merged: #1120 moved The round-trip table keeps its Also checked this against the neighbours: #1110, #1121 and #1127 all touch the same visits, and a tree with all of them merged onto |
nielspardon
left a comment
There was a problem hiding this comment.
Rebase reasoning checks out — visit(IntervalDayLiteral) on main does go through the type converter now, so that half really is subsumed rather than merged. Design verified against Calcite 1.42: makeAbstractCast does no literal folding, so the nullable shape is stable and matches what the scalar visitors produce. Three small things inline.
Separately, the precision_timestamp_tz degradation and the microsecond normalization are recorded only in a test comment — worth issues so they don't get lost.
Calcite types every non-null RexLiteral NOT NULL, so a nullable Substrait literal came back non-nullable and a homogeneous NestedList of a literal and a field reference came back mixed-typed and was rejected. Convert nullable scalar literals as CAST(literal AS nullable type) - the shape the character paths already produced - and fold the nullability-only cast back into a nullable literal on the return trip. LogicalValues tuples keep bare literals: the row type already declares field nullability there.
The tuple unwrap stripped any cast over a literal, so a cast doing real work — a varchar literal longer than its declared length, which Calcite cannot fold — was dropped and the literal landed under a row type that does not describe it. It used to fail loudly there, and does again. The literal fold gets the same treatment for the same reason: an exact no-op cast is not the nullability representation, so it stays a cast. Neither direction can produce one today, since Calcite drops an exact cast over a literal, but the guard now says what the contract says. Plus a test on the cast shape itself, which both the fold and the unwrap key on and nothing pinned.
85a5996 to
30234ee
Compare
|
Rebased. The core commit is gone: #1136 landed the same one-line fix, and its On the two things recorded only in a test comment — both are filed already, and both are closed by #1121: #1114 for the microsecond normalization, whose interval third went with #1120, and #1113 for the |
Substrait literal nullability did not survive conversion to Calcite:
RexBuildergives every non-null literal a NOT NULL type, so nullable literals came back non-nullable, and a homogeneousNestedListholding a nullable literal next to a nullable field reference came back mixed-typed and was rejected. Only the character/binary paths kept the type, incidentally, by picking up a cast wrapper.This makes that shape the contract. Scalar literal visitors request their exact Substrait type (
makeLiteral(value, type, allowCast = true)); themakeTimeLiteral/makeTimestampLiteral/makeIntervalLiteralpaths wrap throughpreserveNullability. Interval-day no longer needs it: #1120 moved that visit onto the type converter, which already carries nullability. On the way back,CallConverters.CASTfolds a cast over a non-null literal that changes nothing but nullability into the literal itself, carrying the cast's nullability, instead of surfacing anExpression.Cast.LogicalValuestuples keep bare literals — the row type already declares field nullability there, so that one cast is unwrapped, and only that one: a cast doing anything else is doing work, and dropping it would leave the literal under a row type that does not describe it.Found along the way and left as is, tracked separately: the Calcite→Substrait path normalizes time/timestamp literals to microsecond precision (#1114) and a
precision_timestamp_tzliteral returns as a plainprecision_timestamp(#1113).Closes #1066
BREAKING CHANGE: Substrait→Calcite conversion now represents a nullable literal as a cast of the literal to its nullable type where it previously produced a bare literal with a non-nullable type, and converted row types become nullable accordingly. Calcite→Substrait conversion folds nullability-only casts over literals into nullable literals instead of Expression.Cast.