Skip to content

Update CSVParser to further increase csv ingest speed - #57

Merged
kenwenzel merged 2 commits into
mainfrom
53-increase-csv-ingest-speed
Sep 7, 2026
Merged

Update CSVParser to further increase csv ingest speed#57
kenwenzel merged 2 commits into
mainfrom
53-increase-csv-ingest-speed

Conversation

@wmehling

@wmehling wmehling commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Update CSV parser for more speed

This builds on the CSV ingestion optimizations from #54 with two additional parser improvements:

  • 1ddfdaf — Replace regex-based CSV double parsing with a guarded Double.parseDouble path, reducing allocations.
  • d853326 — Parse numbers before booleans, avoiding unnecessary lowercase string creation on the common numeric path.

Together with the two previous ingestion optimizations (f3ce7ef, 75c8ba4), this brings postCsv throughput to approximately 1.2M tuples/s.

Benchmarks on my machine

Compared to main, we achieve an additional ~25% improvement with around 1.25M tuples/s.

  • main: postCsv → 1,007,664 tuple/s
  • now_: postCsv → 1,252,745 tuple/s (+ 24,3%)

Before (main on commit 4ce4460 with grouping and prefix reuse)

Benchmark                                       Mode  Cnt        Score        Error  Units
KvinIngestionBenchmark.postCsv                 thrpt   10  1007664.682 ±  69193.628  ops/s
KvinIngestionBenchmark.postCsvSequentialFiles  thrpt   10   915495.113 ± 268659.103  ops/s
KvinIngestionBenchmark.postJson                thrpt   10   664922.719 ±  53218.133  ops/s
KvinIngestionBenchmark.putBatch                thrpt   10  1879005.933 ± 116042.703  ops/s
KvinIngestionBenchmark.putCsvDirect            thrpt   10  1108954.499 ±  69651.111  ops/s

After (with additional csv parser update)

Benchmark                                       Mode  Cnt        Score        Error  Units
KvinIngestionBenchmark.postCsv                 thrpt   10  1252745.033 ±  59247.065  ops/s
KvinIngestionBenchmark.postCsvSequentialFiles  thrpt   10  1158652.413 ± 104261.836  ops/s
KvinIngestionBenchmark.postJson                thrpt   10   637689.914 ±  84441.013  ops/s
KvinIngestionBenchmark.putBatch                thrpt   10  1602847.570 ± 510060.219  ops/s
KvinIngestionBenchmark.putCsvDirect            thrpt   10  1359744.831 ± 258922.661  ops/s

Doubles.tryParse checks every candidate with a regex. Guard Double.parseDouble instead to avoid matcher allocations while preserving CSV value semantics.
Most ingested values are numeric. Parse them first so they do not require a lowercase copy only for boolean detection.
@wmehling wmehling self-assigned this Sep 4, 2026
@wmehling
wmehling marked this pull request as ready for review September 4, 2026 12:18
@wmehling
wmehling requested a review from kenwenzel September 4, 2026 12:18
@kenwenzel
kenwenzel merged commit 2fe223d into main Sep 7, 2026
1 check passed
@wmehling
wmehling deleted the 53-increase-csv-ingest-speed branch September 8, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants