CAS: recoverable Code 210 on ref-log abandon escapes into noexcept afterCommit → std::terminate (Server died)

[alsugiliazova]

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report - something's broken

Describe the situation

On a content-addressed (CAS) disk, a recoverable CAS write refusal (Code 210 NETWORK_ERROR, “retrying later”) thrown from PartWriteTxn::abandon during ContentAddressedTransaction::publishStaging can escape into MergeTreeTransaction::afterCommit / TransactionLog::finalizeCommittedTransaction, both marked noexcept. That calls std::terminate, aborts the server with SIGABRT (6), and the CI job reports Server died with a mass cascade of later test failures.

Discovered on PR #2300 ASan CAS S3 stateless CI. This is not ASAN MemoryTracker / RSS accounting noise: the Fatal window has no memory-limit / OOM / AddressSanitizer report; afterCommit even installs LockMemoryExceptionInThread (blocks tracker faults).

This issue:

• Causes process abort (std::terminate → SIGABRT) on sanitizer CAS S3 lanes under load
• Turns a retryable CAS condition into an unrecoverable server death
• Inflates CI with hundreds of cascade FAILs after the abort
• Is a CAS product bug (deterministic once Code 210 hits this path); ASan load makes the trigger more likely but is not the root cause

---

How to reproduce the behavior

Environment

• Version: 26.6.4.20001.altinityantalya (PR CAS improvements #2300 head f377ba3a499beacf5d032f3d34a69185765b8699, merge d898800…)
• Build type: amd_asan_ubsan, default MergeTree policy = CAS S3
• Triggering test: 01169_old_alter_partition_isolation_stress (transactional partition stress; concurrent INSERTs / ALTER PARTITION / COMMIT)

Steps

1. Run stateless tests with CAS S3 as the default MergeTree storage policy under ASan (as in CI job below), including 01169_old_alter_partition_isolation_stress.
2. Under concurrent CAS load on the same table namespace, a COMMIT that persists txn_version.txt (CSN metadata) hits a CAS pre-attempt refuse on the scratch-build abandon() after repoint.
3. Observe Fatal: Terminate called for uncaught exception with Code 210, then Received signal Aborted (6) on query COMMIT;.

Minimal local repro of the contract bug (once Code 210 is injected on abandon during publishStaging while inside MergeTree COMMIT finalize) should not require the full stress test; the CI path is:

COMMIT
  → TransactionLog::finalizeCommittedTransaction (noexcept)
    → MergeTreeTransaction::afterCommit (noexcept)
      → setAndStoreCreationCSN → write txn_version.txt on CAS part
        → ContentAddressedTransaction::publishStaging (repoint path)
          → PartWriteTxn::abandon()  // precommit-removal append
            → CasRefLedger append refused BEFORE any request (NoAttemptSent)
              → throw NETWORK_ERROR "retrying later"
                → std::terminate

---

Expected behavior

A recoverable CAS Code 210 during scratch-build abandon() / metadata publish should either:

• be caught and logged (as ~ContentAddressedTransaction already does for destructor abandon), and/or
• surface as a normal failed COMMIT / retryable client error before the noexcept barrier,

and must not abort the server.

---

Actual behavior

On sanitizer CAS S3 builds

Server aborts:

Terminate called for uncaught exception:
Code: 210. DB::Exception: CAS write could not be committed
(CAS ref-log append for namespace '…@cas@' txn … was refused BEFORE any request was sent …);
retrying later. (NETWORK_ERROR)

(query: COMMIT;) Received signal Aborted (6)

Relevant stack (trimmed):

Cas::makeCasWriteRetryLaterExceptionPtr
CasRefLedger::commitRefChunk / flushRefBatch / appendRefOps
PartWriteTxn::abandon
ContentAddressedTransaction::publishStaging
MergeTreeTransaction::afterCommit          // noexcept
TransactionLog::finalizeCommittedTransaction  // noexcept
InterpreterTransactionControlQuery::executeCommit

Related log line (destructor path — correctly caught)

Same Code 210 during unwind is logged and swallowed:

ContentAddressedTransaction: abandoning a build during transaction destruction
(a live precommit binding may persist until remount): Code: 210. …

So the destructor already treats abandon failure as non-fatal; publishStaging does not.

---

Root cause analysis

1. afterCommit is noexcept and writes per-part version metadata to disk (VersionMetadataOnDisk::storeInfoToDataPartStorage → txn_version.txt).
2. On CAS, a standalone write on an already-committed part uses the repoint path in ContentAddressedTransaction::publishStaging: scratch precommitAdd → repointRef → abandon() of the scratch build.
3. Source comment at ContentAddressedTransaction.cpp (~376–379) already states that scratch abandon() can itself throw; the code only protects the CommitOutcome slot, not the process.
4. PartWriteTxn::abandon performs a correctness-bearing precommit-removal via appendRefOps. A pre-attempt gate refuse (CasUnresolvedReason::NoAttemptSent — mount fence or operation deadline before the first putIfAbsent) becomes makeCasWriteRetryLaterExceptionPtr (Code 210).
5. That exception leaves publishStaging and hits the noexcept MergeTree txn finalize → std::terminate.

Not MemoryTracker going negative / snapping to RSS under ASAN: no tracker/OOM evidence in the Fatal window; afterCommit blocks memory exceptions via LockMemoryExceptionInThread.

Suggested fix direction

• Catch / absorb retry-later around abandon() in publishStaging (mirror ~ContentAddressedTransaction), and/or
• Ensure MergeTree afterCommit disk writes cannot throw into noexcept (fail the commit earlier, or make the CAS metadata write path noexcept-safe).

---

Additional context

CI failure

• Job: Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)
• PR / Branch: #2300 feature/antalya-26.6/CAS-improvements
• Commit: f377ba3a499beacf5d032f3d34a69185765b8699
• Run: 34388936376
• Job: 102757844698
• CI report: ci_run_report.html

Binary CAS S3 lanes on the same SHA were green; ASan 1/2 was green — load/timing sensitive trigger, hard abort once hit.

Related

• Verification notes: cas/docs/PR-2300-CI-VERIFICATION.md in clickhouse-regression
• Related but distinct: #2343 (ref_catalog/_ckpt Code 210 under ATTACH), #2332 (mount lease under concurrent FINAL)