53 increase csv ingest speed - #54
Conversation
kenwenzel
left a comment
kenwenzel
left a comment
There was a problem hiding this comment.
The CsvIngestionPipeline could in general also be used by other formats and there be generalized. Furthermore, a lock-free communication can be faster especially for smaller batch sizes.
| } | ||
|
|
||
| IExtendedIterator<List<KvinTuple>> groups = parser.parseRowGroupBatches(rowsPerGroup); | ||
| BlockingQueue<Object> queue = new ArrayBlockingQueue<>(queueCapacity); |
There was a problem hiding this comment.
This is slower than using a CircularBuffer and Thread.onSpinWait() that allows to lock-free send data between too threads.
|
@wmehling Do you want to rebase this to main or should I do? |
|
I'll rebase and clean it up a bit so it is ready for review |
bcf3fe6 to
75c8ba4
Compare
|
@wmehling This does currently not include the multi-threading support for parsing and loading the data in parallel. Is this intended? |
|
Yes, I wanted to reduce the size of the PR and only include the methods that speed up, but that are also robust. The other implementation of the multi-threading was not as reliable. So for now this is the base, which achieved around +23% Speed up with your ideas. I will invest some tokens to see, if we can find a fast, but also reliable and simple implementation for the multi-threading and see if that brings enough speed up to justify the complexity that comes with it. I'll keep you posted @kenwenzel once I have reviewed it / happy with it. Will probably be next week though |
|
@wmehling We have also experimented with several multi-threading approaches for RDF4J. The fastest is a lock-free circular buffer as implemented here: |
2 participants
Increased CSV Ingest from ~400k Tokens/s to ~ 800k Tokens/s
Depends on #51 / Builds on branch, so review #52 first
The optimization was split into three commits so each idea remains independently reviewable, benchmarkable, and revertible:
Column-major row groups: CSV rows are collected in bounded groups of 1,000 and emitted column by column. Creates consecutive tuples with the same item/property/context. Throughput increased from the baseline 391.7k/502.7k tuples/s to **528.0k/561.1k tuples/**s in two independent runs.
Prefix and lock reuse: The LevelDB batch writer reuses the resolved prefix and lock for consecutive tuples with the same identity. Throughput increased further to 602.9k/635.9k tuples/s.
Bounded parser/storage overlap: One task parses CSV row groups while the request thread stores them through a bounded two-group queue. Final endpoint throughput reached 851.9k tuples/s in Run A and 694.7k tuples/s in Run B.
The existing row-major parse() API and read path remain unchanged; only the optimized ingestion route uses column-major ordering within each bounded group.