Feature/tsfile dataframe runtime

[ColinLeeo]

TsFileDataFrame Runtime

Problem

When querying a dataset containing multiple TsFiles, the previous path repeatedly opened files, scanned metadata, and assembled results in Python. Query latency and process memory therefore increased with the number of files. Dataset-level schema validation and file-change detection were also incomplete, which could lead to incorrect merges or reads from stale files.

This PR introduces a persistent Dataset Index and a process-local Runtime. Dataset-level routing, schema information, and file generations are materialized once, while multiple processes can share the read-only index pages through mmap.

Core mechanisms

Persistent Dataset Index

• Stores strings, tables, devices, columns, logical series, file records, file spans, and series locators in a versioned 13-section binary format.
• The header and every section contain size, record layout, count, and CRC metadata.
• The index is read through mmap. Name lookup uses a hash followed by string verification, so hash collisions are handled correctly.
• SeriesLocator identifies the corresponding series metadata in a TsFile. The standard TsFile Reader remains responsible for parsing ChunkMeta.
• Index construction validates file fingerprints, complete TableSchema definitions, and timestamp uniqueness.

Process-local Runtime

• Keeps the read-only Dataset Index mapping, table/series catalog, and a bounded ReaderSession LRU pool.
• Uses leases to control Runtime and Reader lifetimes. Closing a DataFrame does not invalidate an independent lease that is still active.
• Revalidates file size and fingerprint when opening a ReaderSession, detecting replacement or modification after index publication.
• Uses generation validation to prevent an old index from routing reads to new file contents.
• Caches prepared query context, expanded series descriptors, and file shards to avoid repeating metadata work.

Query and Python integration

• Adds C++ Dataset Index, prepared-series, and TsBlock reader interfaces.
• Reads aligned series through a shared time axis and supports mixed selections across devices and columns.
• Adds Python support for table model, tree model, aligned timeseries, and non-aligned timeseries.
• Adds Cython typed-merge and aligned-scatter kernels to reduce Python loops and intermediate objects.
• The pure _merge extension no longer links against C++ libtsfile, avoiding dynamic-library lookup issues from the subpackage directory.

Wheel packaging

Linux, macOS, and Windows wheel builds use bundled C++ dependencies:

-Dtsfile.dependency.source=BUNDLED

This prevents macOS wheels from including Homebrew libraries with an incompatible deployment target and prevents Windows wheels from missing LZ4, LZMA, or zlib runtime dependencies.

Compatibility

• Preserves the public TsFile read/write API compatibility targeted at 2.4.0.
• The Dataset Index format is versioned and checksummed; indexes must be rebuilt when the format version changes.
• Tables with the same name but different TableSchema definitions are not silently merged by field union; they are kept as distinct schema variants or reported as conflicts.
• Duplicate timestamps are reported during index construction.
• C++ clients using the extended metadata structures must be rebuilt with the new headers.

Validation

• C++: 760 passed, 1 skipped
• Python: 196 passed
• Bundled macOS C++ build: passed
• git diff --check: passed
• Multi-platform Linux, macOS, and Windows wheel CI triggered.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 63.15789% with 357 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.24%. Comparing base (d4c3c94) to head (33a15fb).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
cpp/src/dataset/dataset_index.cc	66.80%	92 Missing and 64 partials ⚠️
cpp/src/cwrapper/tsfile_cwrapper.cc	5.88%	77 Missing and 3 partials ⚠️
cpp/src/file/tsfile_io_reader.cc	65.17%	13 Missing and 26 partials ⚠️
cpp/src/reader/qds_without_timegenerator.cc	12.50%	34 Missing and 1 partial ⚠️
...src/reader/block/prepared_series_tsblock_reader.cc	73.07%	11 Missing and 17 partials ⚠️
cpp/src/reader/tsfile_series_scan_iterator.cc	85.71%	0 Missing and 8 partials ⚠️
cpp/src/dataset/dataset_index.h	76.47%	4 Missing ⚠️
cpp/src/reader/tsfile_executor.cc	86.20%	0 Missing and 4 partials ⚠️
cpp/src/file/read_file.cc	84.61%	0 Missing and 2 partials ⚠️
cpp/src/reader/tsfile_reader.cc	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #911      +/-   ##
===========================================
+ Coverage    61.21%   61.24%   +0.02%     
===========================================
  Files          740      746       +6     
  Lines        50571    51533     +962     
  Branches      8141     8341     +200     
===========================================
+ Hits         30959    31562     +603     
- Misses       18125    18356     +231     
- Partials      1487     1615     +128     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hongzhi-gao]

The dataset index is always published as .tsfile_dataframe_index.tsidx under the common parent of the input files. Two TsFileDataFrame instances that share a directory but not the same file set will contend for that one path, rebuild on every mismatch, and can clobber each other. Please derive the index path from the file-set identity (or let the caller pass it), and add a same-directory / different-subset test.