Skip to content

fix(isthmus)!: preserve interval_day precision in both conversion directions - #1120

Merged
nielspardon merged 3 commits into
substrait-io:mainfrom
alexandrefimov:issue-1114-interval-precision
Aug 19, 2026
Merged

fix(isthmus)!: preserve interval_day precision in both conversion directions#1120
nielspardon merged 3 commits into
substrait-io:mainfrom
alexandrefimov:issue-1114-interval-precision

Conversation

@alexandrefimov

@alexandrefimov alexandrefimov commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Substrait's interval_day<P> carries a fractional-second precision, and every path across the Calcite boundary discarded it. TypeConverter built every interval from the shared SubstraitTypeSystem.DAY_SECOND_INTERVAL qualifier at precision 6, ExpressionRexConverter had a second private qualifier pinned at 3 — so the two directions of the same conversion disagreed about the type of the same value — and coming back, LiteralConverter reported a fixed 6 whatever the Calcite literal's type said:

interval_day<0> -> INTERVAL DAY TO SECOND(6)  scale=6
interval_day<3> -> INTERVAL DAY TO SECOND(6)  scale=6

Build the qualifier from P instead, in both places, and read the declared precision back off the type when converting a Calcite interval literal. Calcite keeps a qualifier's precision as the type's scale, and TypeConverter.toSubstrait already read that scale, so the round trip becomes identity:

IntervalDayLiteral{days=1, seconds=2, subseconds=3, precision=3}
  -> 86402003:INTERVAL DAY TO SECOND(3)
  -> IntervalDayLiteral{days=1, seconds=2, subseconds=3, precision=3}

An interval written without an explicit precision is unaffected — Calcite gives it scale 6, which is exactly what was hardcoded. One written as DAY TO SECOND(P) parses to that P and now keeps it.

The value stays in milliseconds, which is what Calcite's own interval literals carry. A sub-millisecond component is therefore still truncated; what this changes is that the declared precision is no longer overwritten with a fixed 6.

With P available on the operand, DATETIME_SUBTRACT now infers precision_timestamp<P> exactly, as functions_datetime.yaml declares, instead of falling back to the type system maximum — the gap #1099 had to leave as a best-effort comment. The parameterized test it left behind, which could only vary the declared output, now varies the argument precision too.

Closes #1118, which reports the type-conversion half of this and was filed just before this PR went up.

It also removes the hardcoded 6 on the literal side — LiteralConverter reported intervalDay(..., 6) whatever scale the Calcite literal carried — so the interval third of #1114 goes with it. The TIME / TIMESTAMP halves of #1114 are a separate value rescale, in #1121.

BREAKING CHANGE: an interval_day precision outside 0 to 9 is now rejected instead of silently narrowed to 6. The bound is getMaxScale(INTERVAL_DAY_SECOND), the same one Calcite validates an interval qualifier against, so nothing that parses from SQL is refused. Converted Calcite types and generated SQL now carry the interval's declared precision, so INTERVAL ... DAY TO SECOND(P) replaces the previous fixed DAY TO SECOND(3). A Calcite interval literal whose declared scale is below 3 now converts to a Substrait interval_day at that scale, dropping the millisecond remainder towards zero, where it previously reported interval_day<6> and kept it.

Substrait's interval_day<P> carries a fractional-second precision, but the
conversion to Calcite discarded it: TypeConverter built every interval from the
shared SubstraitTypeSystem.DAY_SECOND_INTERVAL qualifier at precision 6, and
ExpressionRexConverter had a second private qualifier pinned at 3, so the two
directions of the same conversion disagreed about the type of the same value.

Build the qualifier from P instead, in both places, and read the declared
precision back from the type when converting a Calcite interval literal. Calcite
keeps the qualifier's precision as the type's scale, and the reverse conversion
already read that scale, so interval_day<P> now survives a round trip. SQL-parsed
intervals are unaffected: Calcite gives them scale 6, which is what was hardcoded.

The value stays in milliseconds, which is what Calcite's own interval literals
carry, so a sub-millisecond component is still truncated - only the declared
precision is preserved, and that is now what the type says rather than a fixed 6.

With P available on the operand, DATETIME_SUBTRACT infers precision_timestamp<P>
exactly, as the spec declares, instead of falling back to the type system maximum.

BREAKING CHANGE: an interval_day precision above the Calcite type system maximum
(6) is now rejected instead of being silently narrowed, matching how the
precision_time / precision_timestamp / precision_timestamp_tz conversions already
behave. Generated SQL and converted Calcite types now carry the interval's
declared precision, so INTERVAL ... DAY TO SECOND(P) replaces the previous fixed
DAY TO SECOND(3).
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Heads-up on ordering: this conflicts textually with #1110, which wraps the same makeIntervalLiteral call in its new preserveNullability(...) while this changes the qualifier argument. The two compose — the resolution is preserveNullability(rexBuilder.makeIntervalLiteral(value, SubstraitTypeSystem.daySecondInterval(expr.precision())), expr.getType()) — so whichever lands second just needs that one hunk resolved. No behavioural interaction between them.

@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Checked the interaction with the other two rather than assuming: merged #1110, this, and #1121 together on top of main, resolved the one hunk as described above, and the combined tree builds with :core:test 710 and :isthmus:test 1078 green. #1121 merges cleanly on top with no resolution needed.

The resolution, for the record:

    return preserveNullability(
        rexBuilder.makeIntervalLiteral(
            new BigDecimal((expr.days() * MILLIS_IN_DAY + expr.seconds() * 1_000L + milliseconds)),
            SubstraitTypeSystem.daySecondInterval(expr.precision())),
        expr.getType());

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three blocking things: a declared scale below 3 now loses value where the fixed 6 didn't (1), the new guard reads Calcite's leading-field bound instead of the fractional-second one (2), and the literal path skips it entirely (3). Comment 1 is the one I'd want your call on rather than a fix.

Separately, Spark rejects any interval_day precision but 6 (Util.assertMicroseconds), so isthmus-produced plans will hit it more often after this — a follow-up issue, not this PR.

Comment thread isthmus/src/main/java/io/substrait/isthmus/TypeConverter.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/expression/ExpressionRexConverter.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/expression/LiteralConverter.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/expression/FunctionMappings.java Outdated
Comment thread isthmus/src/test/java/io/substrait/isthmus/CalciteLiteralTest.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/SubstraitTypeSystem.java Outdated
Read the fractional-second ceiling from getMaxScale(INTERVAL_DAY_SECOND) rather
than getMaxPrecision(INTERVAL_DAY): for an interval the latter bounds the leading
field, and the type produced carries INTERVAL_DAY_SECOND, which SubstraitTypeSystem
does not override. That is also the knob SqlValidatorImpl checks a qualifier
against, so SQL written as DAY TO SECOND(7) parses today and the previous guard
would have refused to convert such a plan back. The ceiling is 9, and a lower
bound is added: -1 is Calcite's PRECISION_NOT_SPECIFIED and would otherwise
round-trip as 6.

Scale the whole interval value before taking it apart. Duration reports -1500 ms
as -2 seconds plus a positive 500 ms part, so dropping the sub-second component at
a precision below 3 rounded away from zero, turning -1500 ms into -2000 ms.

Build the interval literal's Calcite type from the type converter like every other
literal visit, instead of from a qualifier: that applies the precision bound and
the literal's nullability, which the qualifier path silently dropped.

Bound the inferred timestamp precision from below as well, and say in the comment
that the cap is the datetime ceiling rather than something the argument decides.

Drop the now unreferenced DAY_SECOND_INTERVAL constant.
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

Thanks for this one — the getMaxPrecision / getMaxScale split is not something I would have found on my own, and it turned a guard that looked like a safety net into one that would have rejected plans Isthmus itself emits. I checked every claim before touching anything and they all held, including the -2000 ms I doubted: Duration normalises -1500 ms to seconds=-2 with a positive 500 ms part, so the rounding really does run away from zero.

All eight addressed in 941059b, replies in the threads. The one you left to me went with honouring the declared precision, with the rounding fixed as a separate defect — reasoning in the first thread, happy to be argued out of it.

Filed #1125 for the Spark side.

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things left: the rewritten arithmetic overflows inside the spec's day range, and the #995 reference is mine to take back.

Comment thread isthmus/src/main/java/io/substrait/isthmus/expression/LiteralConverter.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/expression/FunctionMappings.java Outdated
Scaling the whole millisecond value into the declared precision's units
overflows a long past ~292 years at P=9, well inside the [-3,650,000..
3,650,000] day range the spec allows for interval_day. Decomposing in
milliseconds first keeps the towards-zero narrowing with nothing larger
than 999 left to scale.
@alexandrefimov
alexandrefimov force-pushed the issue-1114-interval-precision branch from 8fb856c to 758b5a8 Compare August 19, 2026 06:39

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nielspardon
nielspardon merged commit 741d4bf into substrait-io:main Aug 19, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

isthmus: Substrait→Calcite type conversion pins interval_day<P> to microsecond precision

2 participants