Skip to content

Pipe: Fix TsFile reference races and improve result set diffs - #18376

Open
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-reference-race
Open

Pipe: Fix TsFile reference races and improve result set diffs#18376
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-reference-race

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Description

PipeTsFileResourceManager.increaseFileReference checked whether a resource existed before entering the creation critical section. Historical and realtime extractors could both observe a missing resource, then create and put separate PipeTsFileResource instances for the same pipe file. The later put overwrote the first reference count, so releasing one event could delete a file that another event still needed. In the reported table pattern IT, this stopped realtime transfer and left the t0 row missing.

This PR:

  • rechecks the private resource map after acquiring the segment lock and reuses the resource created by another caller;
  • covers both hard-linked TsFiles and copied sidecar files such as .mods/.resource;
  • rolls back a private reference when creating the shared/public reference fails, preventing a leaked private reference and file;
  • makes integration-test Set<String> assertions print concise missing and unexpected rows instead of two complete, highly repetitive sets (up to 20 rows per side).

The concurrent regression tests start 64 retain operations together and verify that both private and shared reference counts reach 64, then verify complete cleanup after all releases. A separate failure-path test verifies that a failed public reference increase leaves neither a private count nor a copied file behind.

Validation

  • mvn -pl iotdb-core/datanode -Dtest=PipeTsFileResourceManagerTest test (5 tests passed)
  • mvn -pl integration-test -P with-integration-tests -DskipTests test-compile
  • DataNode Checkstyle and Spotless checks
  • Integration-test Checkstyle and Spotless checks
  • git diff --check

This PR has:

  • been self-reviewed.
    • concurrent write
  • added comments explaining the why and intent where needed.
  • added unit tests for concurrent hard-link and copied-file acquisition.
  • added a unit test for partial-reference failure rollback.

Key changed/added classes
  • PipeTsFileResourceManager
  • PipeTsFileResourceManagerTest
  • TestUtils

@Caideyipi
Caideyipi force-pushed the fix/pipe-tsfile-reference-race branch from ea2ab0e to 25057ef Compare August 5, 2026 07:42
@Caideyipi Caideyipi changed the title Pipe: Fix concurrent TsFile reference increases Pipe: Fix TsFile reference races and improve result set diffs Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant