Skip to content

fix: respect volatility for all function expressions - #25511

Open
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:1fannnw/fix-udf-volatility
Open

1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:1fannnw/fix-udf-volatility

Conversation

@1fanwang

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #25504.

Rationale for this change

Repeated calls to a volatile higher-order UDF can reuse the first call's result. Comparing two calls can also return true without invoking the function. Both queries now execute each call.

What changes are included in this PR?

The volatility check reads the declared signature for aggregate, window and higher-order expressions, as it already does for scalar expressions. This includes aggregates used as window functions.

What is the testing strategy for this PR?

The regression registers a counter UDF and runs both queries through the SQL engine. A separate check covers all function kinds, each volatility level, and aliased expressions.

Testing Done

Query Before After
Two counter calls in a projection (0, 0), one invocation (0, 1), two invocations
Equality between two counter calls true, no invocations false, two invocations

I used Rust 1.98.1 on macOS arm64 with the installed macOS 15.4 SDK. From the PR checkout, these commands run the same regression source against the baseline and the fix:

export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX15.4.sdk
export CARGO_BUILD_JOBS=2

git worktree add -b 1fannnw/repro-udf-volatility \
  ../datafusion-before-25504 42f3888ca2fc82da8548dc4f13533af7af9b5a9f
git diff 42f3888ca2fc82da8548dc4f13533af7af9b5a9f HEAD \
  -- datafusion/core/tests/user_defined |
  git -C ../datafusion-before-25504 apply -

cargo test --locked --profile ci \
  --manifest-path ../datafusion-before-25504/Cargo.toml \
  --target-dir target/volatility-before \
  -p datafusion --test user_defined_integration volatility -- --nocapture

cargo test --locked --profile ci \
  --target-dir target/volatility-after \
  -p datafusion --test user_defined_integration volatility -- --nocapture
Raw logs

Before the fix, the projection returned:

+-------+--------+
| first | second |
+-------+--------+
| 0     | 0      |
+-------+--------+
calls=1

The equality comparison returned:

+-------+
| equal |
+-------+
| true  |
+-------+
calls=0

After the fix, the projection returned:

+-------+--------+
| first | second |
+-------+--------+
| 0     | 1      |
+-------+--------+
calls=2

The equality comparison returned:

+-------+
| equal |
+-------+
| false |
+-------+
calls=2
  • Local code review completed

Are there any user-facing changes?

Queries affected by the incorrect volatility check return the results of separate function calls. There is no public API change.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@github-actions github-actions Bot added logical-expr Logical plan and expressions core Core DataFusion crate labels Sep 19, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.38%. Comparing base (42f3888) to head (78cbf21).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25511    +/-   ##
========================================
  Coverage   82.37%   82.38%            
========================================
  Files        1138     1138            
  Lines      434081   434322   +241     
  Branches   434081   434322   +241     
========================================
+ Hits       357588   357814   +226     
- Misses      54864    54875    +11     
- Partials    21629    21633     +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@neilconway

Copy link
Copy Markdown
Contributor

There's also #25513 for the same bug

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expr::is_volatile ignores the volatility of aggregate, window and higher-order functions

3 participants