fix(isthmus)!: preserve interval_day precision in both conversion directions - #1120
Conversation
Substrait's interval_day<P> carries a fractional-second precision, but the conversion to Calcite discarded it: TypeConverter built every interval from the shared SubstraitTypeSystem.DAY_SECOND_INTERVAL qualifier at precision 6, and ExpressionRexConverter had a second private qualifier pinned at 3, so the two directions of the same conversion disagreed about the type of the same value. Build the qualifier from P instead, in both places, and read the declared precision back from the type when converting a Calcite interval literal. Calcite keeps the qualifier's precision as the type's scale, and the reverse conversion already read that scale, so interval_day<P> now survives a round trip. SQL-parsed intervals are unaffected: Calcite gives them scale 6, which is what was hardcoded. The value stays in milliseconds, which is what Calcite's own interval literals carry, so a sub-millisecond component is still truncated - only the declared precision is preserved, and that is now what the type says rather than a fixed 6. With P available on the operand, DATETIME_SUBTRACT infers precision_timestamp<P> exactly, as the spec declares, instead of falling back to the type system maximum. BREAKING CHANGE: an interval_day precision above the Calcite type system maximum (6) is now rejected instead of being silently narrowed, matching how the precision_time / precision_timestamp / precision_timestamp_tz conversions already behave. Generated SQL and converted Calcite types now carry the interval's declared precision, so INTERVAL ... DAY TO SECOND(P) replaces the previous fixed DAY TO SECOND(3).
|
Heads-up on ordering: this conflicts textually with #1110, which wraps the same |
|
Checked the interaction with the other two rather than assuming: merged #1110, this, and #1121 together on top of The resolution, for the record: return preserveNullability(
rexBuilder.makeIntervalLiteral(
new BigDecimal((expr.days() * MILLIS_IN_DAY + expr.seconds() * 1_000L + milliseconds)),
SubstraitTypeSystem.daySecondInterval(expr.precision())),
expr.getType()); |
nielspardon
left a comment
There was a problem hiding this comment.
Three blocking things: a declared scale below 3 now loses value where the fixed 6 didn't (1), the new guard reads Calcite's leading-field bound instead of the fractional-second one (2), and the literal path skips it entirely (3). Comment 1 is the one I'd want your call on rather than a fix.
Separately, Spark rejects any interval_day precision but 6 (Util.assertMicroseconds), so isthmus-produced plans will hit it more often after this — a follow-up issue, not this PR.
Read the fractional-second ceiling from getMaxScale(INTERVAL_DAY_SECOND) rather than getMaxPrecision(INTERVAL_DAY): for an interval the latter bounds the leading field, and the type produced carries INTERVAL_DAY_SECOND, which SubstraitTypeSystem does not override. That is also the knob SqlValidatorImpl checks a qualifier against, so SQL written as DAY TO SECOND(7) parses today and the previous guard would have refused to convert such a plan back. The ceiling is 9, and a lower bound is added: -1 is Calcite's PRECISION_NOT_SPECIFIED and would otherwise round-trip as 6. Scale the whole interval value before taking it apart. Duration reports -1500 ms as -2 seconds plus a positive 500 ms part, so dropping the sub-second component at a precision below 3 rounded away from zero, turning -1500 ms into -2000 ms. Build the interval literal's Calcite type from the type converter like every other literal visit, instead of from a qualifier: that applies the precision bound and the literal's nullability, which the qualifier path silently dropped. Bound the inferred timestamp precision from below as well, and say in the comment that the cap is the datetime ceiling rather than something the argument decides. Drop the now unreferenced DAY_SECOND_INTERVAL constant.
|
Thanks for this one — the All eight addressed in 941059b, replies in the threads. The one you left to me went with honouring the declared precision, with the rounding fixed as a separate defect — reasoning in the first thread, happy to be argued out of it. Filed #1125 for the Spark side. |
nielspardon
left a comment
There was a problem hiding this comment.
Two things left: the rewritten arithmetic overflows inside the spec's day range, and the #995 reference is mine to take back.
Scaling the whole millisecond value into the declared precision's units overflows a long past ~292 years at P=9, well inside the [-3,650,000.. 3,650,000] day range the spec allows for interval_day. Decomposing in milliseconds first keeps the towards-zero narrowing with nothing larger than 999 left to scale.
8fb856c to
758b5a8
Compare
Substrait's
interval_day<P>carries a fractional-second precision, and every path across the Calcite boundary discarded it.TypeConverterbuilt every interval from the sharedSubstraitTypeSystem.DAY_SECOND_INTERVALqualifier at precision 6,ExpressionRexConverterhad a second private qualifier pinned at 3 — so the two directions of the same conversion disagreed about the type of the same value — and coming back,LiteralConverterreported a fixed 6 whatever the Calcite literal's type said:Build the qualifier from
Pinstead, in both places, and read the declared precision back off the type when converting a Calcite interval literal. Calcite keeps a qualifier's precision as the type's scale, andTypeConverter.toSubstraitalready read that scale, so the round trip becomes identity:An interval written without an explicit precision is unaffected — Calcite gives it scale 6, which is exactly what was hardcoded. One written as
DAY TO SECOND(P)parses to thatPand now keeps it.The value stays in milliseconds, which is what Calcite's own interval literals carry. A sub-millisecond component is therefore still truncated; what this changes is that the declared precision is no longer overwritten with a fixed 6.
With
Pavailable on the operand,DATETIME_SUBTRACTnow infersprecision_timestamp<P>exactly, asfunctions_datetime.yamldeclares, instead of falling back to the type system maximum — the gap #1099 had to leave as a best-effort comment. The parameterized test it left behind, which could only vary the declared output, now varies the argument precision too.Closes #1118, which reports the type-conversion half of this and was filed just before this PR went up.
It also removes the hardcoded 6 on the literal side —
LiteralConverterreportedintervalDay(..., 6)whatever scale the Calcite literal carried — so the interval third of #1114 goes with it. TheTIME/TIMESTAMPhalves of #1114 are a separate value rescale, in #1121.BREAKING CHANGE: an
interval_dayprecision outside 0 to 9 is now rejected instead of silently narrowed to 6. The bound isgetMaxScale(INTERVAL_DAY_SECOND), the same one Calcite validates an interval qualifier against, so nothing that parses from SQL is refused. Converted Calcite types and generated SQL now carry the interval's declared precision, soINTERVAL ... DAY TO SECOND(P)replaces the previous fixedDAY TO SECOND(3). A Calcite interval literal whose declared scale is below 3 now converts to a Substraitinterval_dayat that scale, dropping the millisecond remainder towards zero, where it previously reportedinterval_day<6>and kept it.