Skip to content

fix: handle null lists with empty children in Spark's shuffle - #25544

Open
neilconway wants to merge 1 commit into
apache:mainfrom
neilconway:neilc/fix-shuffle-null-list
Open

neilconway wants to merge 1 commit into
apache:mainfrom
neilconway:neilc/fix-shuffle-null-list

Conversation

@neilconway

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • N/A

Rationale for this change

The Spark-compatible shuffle implementation unnecessarily copied the first child element as a placeholder for each null list. This is wrong for typed-NULL lists with no children, and was redundant work anyway: NULL lists can be adequately represented by a repeated offset entry, along with setting the appropriate bit in the validity buffer.

What changes are included in this PR?

  • Fix bug
  • Add unit test

What is the testing strategy for this PR?

Existing tests pass, new test added.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Sep 20, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.76923% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.39%. Comparing base (24a7f13) to head (1bbcae1).
⚠️ Report is 19 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/spark/src/function/array/shuffle.rs 80.76% 1 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             main   #25544     +/-   ##
=========================================
  Coverage   82.38%   82.39%             
=========================================
  Files        1138     1138             
  Lines      433739   434746   +1007     
  Branches   433739   434746   +1007     
=========================================
+ Hits       357338   358203    +865     
- Misses      54847    54855      +8     
- Partials    21554    21688    +134     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants