Skip to content

fix(workflow-operator, v1.2): File Scan operator using offset with an empty limit emits no rows - #7530

Merged
aglinxinyuan merged 2 commits into
apache:release/v1.2from
eugenegujing:backport/file-scan-offset-overflow-v1.2
Aug 13, 2026
Merged

fix(workflow-operator, v1.2): File Scan operator using offset with an empty limit emits no rows#7530
aglinxinyuan merged 2 commits into
apache:release/v1.2from
eugenegujing:backport/file-scan-offset-overflow-v1.2

Conversation

@eugenegujing

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Backport of #7348 to release/v1.2, cherry-picked from main commit 8982803.

FileScanUtils.createTuplesFromFile computed the end of its line slice as offset + limit.getOrElse(Int.MaxValue). With Offset ≥ 1 and Limit left empty, the addition overflows Int to a negative bound, and Iterator.slice clamps a negative bound to 0 and returns an empty iterator. The File Scan operator therefore emitted zero rows, silently, with the workflow reporting success. Both FileScan and FileScanOp delegate to this helper, so both were affected.

The fix replaces the slice arithmetic with drop(offset) plus an optional take(limit) — the shape CSVScanSourceOpExec and ArrowSourceOpExec already use — so "no limit" is expressed by not bounding the iterator rather than by a sentinel value that arithmetic can overflow. FileScanUtils.scala on release/v1.2 is byte-identical to main's pre-fix version, so the fix applies unchanged.

One adaptation was needed: FileScanUtilsSpec.scala was created on main (#6077) after v1.2 branched, so the cherry-pick hit a modify/delete conflict; this PR adds the file with main's full post-fix content. As a side effect, v1.2 also gains the spec's 3 pre-existing main-only tests (zip extraction, __MACOSX filtering, per-line flat-map) in addition to the 8 fix-related ones. They target FileScanUtils behavior that is identical on v1.2 and all pass.

Any related issues, documentation, discussions?

Backport of #7348 (originally closed #7345).

How was this PR tested?

The regression tests from #7348 come along with the cherry-pick. On this branch:

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.file.FileScanUtilsSpec org.apache.texera.amber.operator.source.scan.file.FileScanSourceOpDescSpec org.apache.texera.amber.operator.source.scan.file.FileScanOpDescSpec"
# 3 suites, 25 tests, all passed
# (main has 29: four getPhysicalOp/propagateSchema coverage tests were added
#  to these specs after v1.2 branched and are unrelated to this fix)

sbt "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck"
# passed

sbt "WorkflowOperator/scalafixAll --check"
# passed

Was this PR authored or co-authored using generative AI tooling?

Co-authored by: Claude Code (Claude Fable 5)

… empty limit emits no rows

(cherry picked from commit 8982803)
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@aglinxinyuan
aglinxinyuan enabled auto-merge August 12, 2026 05:04
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.83%. Comparing base (9ead9e9) to head (cff9c94).

Additional details and impacted files
@@                Coverage Diff                 @@
##             release/v1.2    #7530      +/-   ##
==================================================
- Coverage           52.84%   52.83%   -0.01%     
  Complexity           2522     2522              
==================================================
  Files                1078     1078              
  Lines               42367    42366       -1     
  Branches             4565     4565              
==================================================
- Hits                22387    22386       -1     
  Misses              18662    18662              
  Partials             1318     1318              
Flag Coverage Δ *Carryforward flag
access-control-service 64.35% <ø> (ø)
agent-service 34.36% <ø> (ø) Carriedforward from 9ead9e9
amber 52.89% <ø> (-0.01%) ⬇️ Carriedforward from 9ead9e9
computing-unit-managing-service 1.65% <ø> (ø)
config-service 56.06% <ø> (ø)
file-service 61.72% <ø> (ø) Carriedforward from 9ead9e9
frontend 47.29% <ø> (ø) Carriedforward from 9ead9e9
pyamber 93.07% <ø> (ø) Carriedforward from 9ead9e9
python 90.88% <ø> (ø) Carriedforward from 9ead9e9
workflow-compiling-service 58.69% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Aug 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 12, 2026
@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Aug 13, 2026
Merged via the queue into apache:release/v1.2 with commit 80de40f Aug 13, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants