fix(workflow-operator): MAX aggregation checks the wrong empty-group sentinel - #7532
fix(workflow-operator): MAX aggregation checks the wrong empty-group sentinel#7532eugenegujing wants to merge 1 commit into
Conversation
…sentinel Change maxAgg's finaliser in AggregationOperation to test the minValue sentinel it initialises with (it tested maxValue, copied from minAgg), and add regression tests for the empty, all-null, and type-maximum cases across INTEGER, LONG, DOUBLE, and TIMESTAMP.
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7532 +/- ##
============================================
+ Coverage 84.70% 85.23% +0.53%
- Complexity 4153 4224 +71
============================================
Files 1169 1173 +4
Lines 46740 46808 +68
Branches 5202 5212 +10
============================================
+ Hits 39592 39899 +307
+ Misses 5433 5162 -271
- Partials 1715 1747 +32
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| ⚪ | bs=10 sw=10 sl=64 | 503 | 0.307 | 18,397/29,294/29,294 us | ⚪ within ±5% / 🔴 +76.3% |
| 🔴 | bs=100 sw=10 sl=64 | 1,040 | 0.635 | 95,296/119,124/119,124 us | 🔴 +9.9% / 🟢 +10.7% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,250 | 0.763 | 807,214/861,629/861,629 us | ⚪ within ±5% / 🟢 +29.4% |
Baseline details
Latest main 133da7b from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 503 tuples/sec | 525 tuples/sec | 740.3 tuples/sec | -4.2% | -32.1% |
| bs=10 sw=10 sl=64 | MB/s | 0.307 MB/s | 0.321 MB/s | 0.452 MB/s | -4.4% | -32.1% |
| bs=10 sw=10 sl=64 | p50 | 18,397 us | 18,108 us | 13,173 us | +1.6% | +39.7% |
| bs=10 sw=10 sl=64 | p95 | 29,294 us | 27,921 us | 16,615 us | +4.9% | +76.3% |
| bs=10 sw=10 sl=64 | p99 | 29,294 us | 27,921 us | 19,272 us | +4.9% | +52.0% |
| bs=100 sw=10 sl=64 | throughput | 1,040 tuples/sec | 1,093 tuples/sec | 939.86 tuples/sec | -4.8% | +10.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.635 MB/s | 0.667 MB/s | 0.574 MB/s | -4.8% | +10.7% |
| bs=100 sw=10 sl=64 | p50 | 95,296 us | 90,323 us | 106,062 us | +5.5% | -10.2% |
| bs=100 sw=10 sl=64 | p95 | 119,124 us | 108,354 us | 113,212 us | +9.9% | +5.2% |
| bs=100 sw=10 sl=64 | p99 | 119,124 us | 108,354 us | 123,853 us | +9.9% | -3.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,250 tuples/sec | 1,253 tuples/sec | 966.11 tuples/sec | -0.2% | +29.4% |
| bs=1000 sw=10 sl=64 | MB/s | 0.763 MB/s | 0.765 MB/s | 0.59 MB/s | -0.3% | +29.4% |
| bs=1000 sw=10 sl=64 | p50 | 807,214 us | 794,502 us | 1,038,733 us | +1.6% | -22.3% |
| bs=1000 sw=10 sl=64 | p95 | 861,629 us | 885,223 us | 1,085,289 us | -2.7% | -20.6% |
| bs=1000 sw=10 sl=64 | p99 | 861,629 us | 885,223 us | 1,115,555 us | -2.7% | -22.8% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,397.51,200,128000,503,0.307,18396.94,29294.22,29294.22
1,100,10,64,20,1922.23,2000,1280000,1040,0.635,95296.05,119124.20,119124.20
2,1000,10,64,20,16000.08,20000,12800000,1250,0.763,807214.20,861629.15,861629.15
What changes were proposed in this PR?
One-line fix in
AggregationOperation.scala: themaxaggregation starts its running maximum from the type's minimum value, so its final "was this group empty?" check must compare against that same minimum value, but it compared against the type's maximum value instead (the line was copied frommin, where that comparison is correct becauseminstarts from the maximum).Because the check looked at the wrong sentinel, two results were silently wrong while the workflow completed with no error:
null, e.g.-2147483648for INTEGER or1970-01-01 00:00:00for TIMESTAMP.maxover{1, 5, 2147483647}reported5(the largest value from the other local aggregation worker) instead of2147483647.Before-and-after
Test data: group
g1has only nulls,g2contains{1, 5, 2147483647},g3contains{10, 42}.Before the fix,
max(v)grouped bykreturns-2147483648forg1(expectednull) and5forg2(expected2147483647):min(v)on the same data is correct (nullforg1,1forg2,10forg3), confirming onlymax's empty-group check is broken:After the fix,
max(v)returnsnullforg1,2147483647forg2, and42forg3:Any related issues, documentation, discussions?
Closes #7531
The regression was introduced by #1840, which generalised the hard-coded
Doublesentinels to per-typeminValue/maxValuehelpers and updatedmaxAgg's initialiser but not its finaliser.How was this PR tested?
Added 5 regression tests, all of which fail without the one-line fix and pass with it (verified in both directions):
AggregateOpSpec:maxover empty input and over all-null input returnsnull;maxkeeps a true maximum equal to the type's maximum value, covering INTEGER, LONG, DOUBLE, and TIMESTAMP.AggregationOperationSpec: a worker-to-final pipeline viagetFinalkeepsInt.MaxValuewhen partial results are re-aggregated, reproducing the two-worker scenario shown above;max's merge stays neutral when one side saw no values.Full aggregate suite:
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.aggregate.*"— 68 tests, all passing.Also verified end to end in the UI with the workflow shown above (screenshots are from before and after rebuilding the backend with this fix).
Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)